Scripting API bulk export truncates content at 4000 chars

General Discussion
1

Scripting API bulk export truncates content at 4000 chars

Glue job chokes on the JSON payload from /api/v2/analytics/details. PySpark stage fails immediately upon deserialization. Content field gets chopped mid-sentence. Logs show the truncation happens exactly at the character limit. Redshift COPY throws validation errors because the structure breaks.

{"error": "JSONDecodeError", "msg": "Unterminated string starting at line 1 column 4002"}

S3 landing zone is full of half-baked files. Don’t see a chunking param in the docs.

2

Does the current capacity model support unbounded JSON payloads, or is there a governance policy restricting export sizes? That’s a hard limit on the serialization buffer. The engineering team should review the payload size constraints and adjust the export configuration. Logs show ...truncated.

3

JSONDecodeError: Unterminated string starting at: line 1 column 4001.

  • The details endpoint caps the inline response at 4k characters.
  • You’ll need to use the async export flow to grab the full payload.
  • Poll the query ID instead of parsing the direct response.
response = platformClient.analyticsApi.postAnalyticsDetailsQuery(body=queryBody)
queryId = response.body.id

Just poll the ID.

4
  • The inline endpoint caps at four kilobytes, so you’ll hit that parse error every time. Switch to the async flow instead. You send the query via postAnalyticsDetailsQuery, grab the returned id, then poll /api/v2/analytics/details/queries/{queryId} until the status flips to complete.
  • Your PySpark stage just needs to fetch the result_url from that status object. The full payload downloads without truncation.
query = platformClient.analyticsApi.getAnalyticsDetailsQuery(query_id)
print(query.body.result_url)
  • The buffer clears once you stop parsing the raw response directly. Just let the queue handle the heavy lifting.
5

HTTP 400 Bad Request: Malformed JSON. The inline endpoint hard-caps responses at four kilobytes to prevent gateway timeouts. You’re smashing that buffer before the async queue spins up.

Poll the query ID instead. The result_url drops a raw file without the event envelope, so your schema registry will choke if you don’t map it.

res = platformClient.analyticsApi.getAnalyticsDetailsQuery(query_id=query_id)

Check the headers first.