I run a script daily to bulk export recordings for legal discovery using POST /api/v2/recording/jobs.
My script is very careful to stay under the standard 300 requests per minute API rate limit. However, whenever I try to submit more than 2 or 3 large export jobs in quick succession, I immediately receive a 429 ‘Too Many Requests’ error on the recording/jobs endpoint.
It seems like there is a separate, hidden limit on how many concurrent bulk export jobs can be running or queued simultaneously in a single org. Can anyone confirm what the actual concurrent job limit is, so I can adjust my script’s backoff logic?
This caught us out too during our gamification reporting!
The Bulk Export API is an asynchronous background service, so it has much stricter concurrency limits than standard synchronous APIs to protect the platform’s S3 bandwidth. While the standard API rate limit is 300/min, the recording/jobs endpoint specifically limits you to 2 concurrent ‘Processing’ jobs per organization. If you try to submit a 3rd job while 2 are already running, you will get hit with that 429.
To fix your script, you cannot just use a simple time-based sleep. You must implement a polling loop. After you submit your first 2 jobs, your script needs to loop and call GET /api/v2/recording/jobs/{jobId} to check their status. Only when one of those jobs transitions to the FULFILLED or FAILED state should your script submit the next POST request for the 3rd job. It’s a true queue management problem, not just a rate limiting problem.
If you have a massive backlog of recordings to export (like millions of files), don’t try to cram them into dozens of small jobs to work around the concurrency limit. It is much more efficient to submit one massive job. A single recording bulk export job can handle up to 100,000 conversation IDs in its payload. Submit one massive 100k job, wait for it to finish, and then submit the next. It reduces the overhead significantly.
That polling loop suggestion is spot on, and I have implemented a similar pattern in our AWS infrastructure to handle these exact 429 errors. Since I manage the backend integrations for our Genesys Cloud setup, I wanted to share a concrete implementation using AWS Lambda Data Actions that might help others facing this bottleneck.
Instead of a simple sleep in a script, I use an EventBridge rule to trigger a Lambda function. The Lambda uses the Genesys Cloud SDK to check the status of existing jobs. If the count of jobs in Processing state is less than 2, it submits the next batch. This decouples the trigger from the execution and handles retries gracefully.
Here is a simplified snippet of the logic inside the Lambda handler:
import gen_rest_client
def check_and_submit():
client = gen_rest_client.create_client()
# Get all jobs
jobs = client.get_recording_jobs()
# Count active processing jobs
active_count = sum(1 for j in jobs if j.status == 'processing')
if active_count < 2:
# Safe to submit new job
submit_new_export_job()
return {"status": "submitted"}
else:
# Wait and retry via EventBridge schedule
return {"status": "waiting", "active_jobs": active_count}
This approach ensures we never hit the hard concurrency limit. It also integrates nicely with our S3 recording export workflows, as we can trigger the next stage of our pipeline only when the job reaches FULFILLED status. It adds a bit of initial setup complexity, but it is far more robust than a fragile cron job.