Hey everyone,
So we’re hitting a weird snag with a Data Action trying to post custom metrics to our DogStatsD instance - intermittent 429s. It’s not constant, which makes it harder to pin down. It’s not a rate limit on our end - we’ve checked that. We’re on Genesys Cloud, naturally.
The flow is pretty simple. Architect flow triggers a Data Action which makes a POST request to https://metrics.example.com/api/v2/employeeperformance/externalmetrics/data. The request body is a JSON string like this:
{
"metric": "call.duration",
"value": 120,
"tags": ["queue:support", "agent:1234"]
}
We’ve set up retry logic within the Data Action itself - a 5-second backoff with three retries - thinking it’s a temporary blip. But it’s still happening, and occasionally the flow fails completely. We’re using the latest Data Action SDK version - 1.2.3.
Now, why are we getting 429s when we think we’ve accounted for potential throttling? That’s the question. Option A is that Genesys Cloud itself is applying some sort of hidden rate limit to Data Actions, even if we’re respecting the documented limits. Seems unlikely, but it’s worth considering. We could try spreading out the calls more - maybe add a 10-second delay before the Data Action even fires? That feels… hacky, though.
Option B is there’s an issue with the API gateway before DogStatsD, and the 429s are a symptom of a larger problem. Our networking team is looking into that, but so far they haven’t found anything.
Here’s a sample error log from the Data Action executions. It’s a bit noisy, but you can see the intermittent 429s:
2024-02-29T14:35:00Z - DataAction execution started
2024-02-29T14:35:01Z - POST request to https://metrics.example.com/api/v2/employeeperformance/externalmetrics/data initiated
2024-02-29T14:35:01Z - Received response: 429 Too Many Requests
2024-02-29T14:35:06Z - Retry attempt 1/3
2024-02-29T14:35:06Z - POST request to https://metrics.example.com/api/v2/employeeperformance/externalmetrics/data initiated
2024-02-29T14:35:07Z - Received response: 200 OK
2024-02-29T14:35:00Z - DataAction execution completed successfully
2024-02-29T15:10:00Z - DataAction execution started
2024-02-29T15:10:01Z - POST request to https://metrics.example.com/api/v2/employeeperformance/externalmetrics/data initiated
2024-02-29T15:10:01Z - Received response: 429 Too Many Requests
2024-02-29T15:10:06Z - Retry attempt 1/3
2024-02-29T15:10:06Z - POST request to https://metrics.example.com/api/v2/employeeperformance/externalmetrics/data initiated
2024-02-29T15:10:06Z - Received response: 429 Too Many Requests
2024-02-29T15:10:11Z - Retry attempt 2/3
2024-02-29T15:10:11Z - POST request to https://metrics.example.com/api/v2/employeeperformance/externalmetrics/data initiated
2024-02-29T15:10:11Z - Received response: 429 Too Many Requests
2024-02-29T15:10:16Z - Retry attempt 3/3
2024-02-29T15:10:16Z - POST request to https://metrics.example.com/api/v2/employeeperformance/externalmetrics/data initiated
2024-02-29T15:10:16Z - Received response: 500 Internal Server Error
2024-02-29T15:10:16Z - DataAction execution failed
See that 500 on the last retry? That’s… lovely!! Any thoughts on where to even start looking here?