I’m building a daily export job in Python to pull interaction analytics and push them to an S3 bucket. The goal is to get all interactions from the previous day, process the JSON, and upload a CSV.
Using the genesyscloud Python SDK. I’m iterating through the results using next_page until it returns None. The issue is that for dates with high volume, the query seems to stop returning results after about 10,000 records, even though I know there are more. The next_page call just returns an empty list or None prematurely.
Here is the loop structure:
from genesyscloud import analytics_api
import boto3
analytics = analytics_api.AnalyticsApi(api_client)
body = {
"query": {
"date_from": "2023-10-27T00:00:00.000Z",
"date_to": "2023-10-27T23:59:59.999Z",
"size": 10000
}
}
all_interactions = []
resp = analytics.post_analytics_interactions_query(body=body)
while resp is not None:
all_interactions.extend(resp.entities)
if resp.next_page is not None:
resp = analytics.get_analytics_interactions_query_by_id(resp.next_page)
else:
break
# Upload to S3
s3_client = boto3.client('s3')
s3_client.put_object(Bucket='my-bucket', Key='export.csv', Body=csv_data)
The size parameter is set to 10,000. The docs say the max is 10,000. But if the total count is 50,000, shouldn’t the pagination handle it? It feels like the next_page token is invalid or the API is silently truncating.
Is there a different endpoint for large datasets? Or am I missing a parameter in the body? I’ve tried increasing the size to 20,000 but it just throws a 400 error.
Any ideas on how to handle this without hitting the wall?