We’ve hooked up Genesys Cloud EventBridge to our internal tracing pipeline. The goal is simple: capture conversation events, extract the traceId from the payload, and create a span in our OTel collector.
Here’s the issue. We’re seeing duplicate events for the same conversation update. Not always, but enough to mess up our metrics. When I look at the raw JSON coming from the Lambda trigger, the detail object is identical. Even the eventID in the outer wrapper is sometimes the same for what looks like two separate triggers.
{
"version": "0",
"id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"detail-type": "Conversation Event",
"source": "com.nice.ccxone",
"account": "123456789012",
"time": "2023-10-27T08:15:00Z",
"region": "us-east-1",
"resources": ["arn:aws:genesys:..."],
"detail": {
"conversationId": "conv-12345",
"eventType": "call.start",
"traceContext": "00-abc123-def456-01"
}
}
The id field in the EventBridge envelope is supposed to be unique per event. It isn’t always. Or maybe Genesys is re-sending the same event on retry and EventBridge is treating it as a new delivery? Our Lambda function is idempotent in theory, but we’re counting spans based on receipt. If we get two identical payloads, we get two spans. That’s bad for our distributed tracing accuracy.
I’ve tried using the traceContext + conversationId as a composite key in DynamoDB for deduplication before creating the span. It works, but it adds latency. Is there a better way?
Should we be looking at the detail-type and time fields to filter? Or is this a known issue with the CXone EventBridge integration where retries aren’t marked as duplicates? I’ve checked the docs but nothing mentions deduplication IDs specifically for GC events. Any code examples for handling this in Node.js?