I’m trying to pull conversation details for the last 24 hours using the Python SDK. The docs mention cursor-based pagination, but the behavior seems inconsistent when I hit the rate limits or when the data is sparse.
I’ve set up a loop to fetch pages. Here’s the core logic:
client = PlatformClientV2(configuration)
analytics_api = client.analytics_api
query = {
"interval": "2023-10-27T00:00:00.000Z/2023-10-28T00:00:00.000Z",
"size": 1000,
"groupBy": "conversationId"
}
cursor = None
while True:
try:
response = analytics_api.post_analytics_conversations_details_query(query, cursor=cursor)
print(f"Got {len(response.entities)} entities")
for entity in response.entities:
cess_entity(entity)
if not response.nextPageCursor:
break
cursor = response.nextPageCursor
time.sleep(0.5) # Trying to be nice to the API
except Exception as e:
print(f"Error: {e}")
break
The issue is that sometimes nextPageCursor is returned, but the next call returns an empty entities array and no new cursor. Other times, it loops indefinitely because the cursor doesn’t change but nextPageCursor is still present.
I’ve checked the raw JSON response. When it stops, the JSON looks like this:
{
"pageSize": 1000,
"pageNumber": 1,
"pageCount": 1,
"total": 0,
"entities": [],
"nextPageCursor": "some_long_string_here",
"previousPageCursor": null
}
It’s weird. If total is 0, why is there a nextPageCursor? I’m assuming this is a bug in the SDK or the API handling empty pages. I’ve tried resetting the cursor manually, but that just restarts the query.
Has anyone else seen this? Is there a way to force a stop if entities is empty but a cursor exists? I don’t want to hard-code a max loop count because that feels brittle. The timezone is Sydney, so I’m dealing with UTC offsets in the interval, but I’ve double-checked the timestamps. They look correct.
The API docs say “cursor-based pagination is supported”, but they don’t mention this edge case. I’m running out of ideas. The sleep helps with rate limiting, but it doesn’t fix the logic loop.
Any thoughts on how to handle this cleanly? I’ve tried catching the empty list and breaking, but I’m worried I’m missing data. The conversation IDs are unique, so duplicates aren’t an issue, but the infinite loop is a blem.