What is the correct way to handle 429s during high-concurrency outbound dialing?

General Discussion
1

Background

Our team is preparing for a major campaign launch next week. We are using Genesys Cloud Outbound for predictive dialing. The goal is to reach 5,000 unique contacts per hour during peak hours. To validate our infrastructure and API limits, we are running load tests using JMeter. The test environment is a US-East production tenant. We are using the standard REST API endpoints for campaign configuration and contact import. The JMeter script is set to ramp up to 200 concurrent threads, simulating agents logging in and starting campaigns simultaneously.

Issue

When the load test reaches approximately 150 concurrent requests to POST /api/v2/outbound/campaigns/{id}/contacts, we start seeing a high rate of HTTP 429 Too Many Requests errors. This happens even though we are well below the documented rate limits for the tenant tier. The error response body indicates that the rate limit has been exceeded for the specific endpoint. However, the Retry-After header is not always present or is inconsistent. Some requests fail immediately, while others are delayed. This causes the JMeter test to fail, as the script does not handle the backoff strategy effectively. We need to ensure that our actual production traffic will not suffer from these throttling issues during peak load.

Troubleshooting

We have checked the following:

  • Verified the rate limits in the developer portal for our tenant tier.
  • Confirmed that the API keys used in JMeter have the correct permissions (outbound:campaign:write).
  • Checked the Genesys Cloud logs for any server-side errors, but only see 429s.
  • Tried adding a fixed delay between requests in JMeter, but this reduces the throughput too much.
  • Reviewed the X-RateLimit-Remaining header, but the values fluctuate unpredictably.

Is there a recommended best practice for handling 429 errors in high-concurrency outbound dialing scenarios? Should we implement a custom backoff algorithm, or is there a way to increase the rate limits for our tenant? Any advice on configuring JMeter to handle these errors gracefully would be appreciated.

2

This is caused by hitting the global rate limit for campaign updates. You need to implement exponential backoff in your JMeter script and stagger requests to stay under the 10 requests per second threshold for /api/v2/outbound/campaigns.