Trying to understand why my pagination loop is returning duplicate data sets.
429 Too Many Requests (Rate Limit)
I’m building a custom ingestion job to pull interaction/voice analytics into my data lake. I have the OAuth token refresh handled via a background thread, so auth isn’t the issue here. The problem is strictly with the paging object behavior. I’m using the Python SDK get_analytics_interactions_query.
Here is the loop logic:
current_page = 1
page_size = 100
while True:
response = client.get_analytics_interactions_query(
query_body=query_params,
page_number=current_page,
page_size=page_size
)
# Process records
for record in response.records:
kafka_producer.send(record)
# Check for more pages
if current_page >= response.page_count:
break
current_page += 1
The response.page_count returns 5 initially. I pull pages 1-5. When I loop back to check for new data, page_count is still 5, but the records in page 1 are identical to what I just processed. I assumed pageNumber is purely offset-based, but it seems like the API might be caching the query result set for a specific duration or there’s a subtle constraint on how pageSize interacts with the query window. Is there a recommended pattern for incremental pulls using this API, or am I misinterpreting how pageCount is calculated relative to the timeInterval?