Encountering a persistent data integrity issue when executing bulk export jobs for digital channels (specifically WhatsApp and Web Chat) within the London region environment (v2024.02). The objective is to retrieve complete message transcripts along with full metadata headers for ongoing legal discovery requests. Chain of custody requirements mandate that every message payload includes the legal_hold_status, original_timestamp, and agent_id fields without exception.
The current implementation uses the Bulk Export API to trigger jobs targeting the last 30 days of interaction data. While the initial job status returns COMPLETED_SUCCESS, a subsequent audit of the downloaded JSON files from the designated S3 bucket reveals significant gaps. Approximately 15% of the exported records are missing the legal_hold_status field entirely. Furthermore, several records show a timestamp drift of up to 400 milliseconds compared to the source system logs, which compromises the chronological integrity required for litigation support.
Investigation into the API response headers indicates no immediate errors during the export trigger phase. However, inspecting the raw JSON structure of the affected files shows that the metadata object is often truncated or nullified for messages involving handoffs between digital agents and human supervisors. This truncation appears to occur specifically when the message size exceeds 2KB or when special characters are present in the WhatsApp payload.
Has anyone encountered similar metadata truncation issues with the Bulk Export API for digital channels? Is there a known limitation regarding payload size or character encoding that causes the export engine to drop legal hold metadata? Alternatively, is there a specific configuration parameter or header required in the export request to force the inclusion of complete audit trail metadata, regardless of message length? Any insights into resolving this timestamp drift or ensuring 100% metadata retention would be greatly appreciated, as the current data gap poses a serious compliance risk.