How do I reliably deduplicate events from Genesys Cloud EventBridge without dropping the trace context?
I’m seeing duplicate conversation:participant:updated events hitting my consumer endpoint. The timestamps are identical, and the payload hashes match exactly. This is messing up my distributed tracing pipeline because I’m creating a new span for every single event, leading to massive fan-out in Jaeger.
I tried using the event_id in the payload as a deduplication key in my local cache, but the volume is high enough that I’m hitting memory limits. Plus, I need to ensure that the traceparent header attached to the original webhook request is preserved for the deduplicated events so the trace chain doesn’t break.
Here’s a snippet of my current handler:
import os
from opentelemetry import trace
from opentelemetry.propagate import inject, extract
tracer = trace.get_tracer(__name__)
def handle_event(event):
# Extract context from incoming headers
ctx = extract(event['headers'])
# Check cache for dedup
event_id = event['body']['event_id']
if is_duplicate(event_id):
return {'statusCode': 200}
with tracer.start_as_current_span("process-gc-event", context=ctx) as span:
span.set_attribute("event.id", event_id)
# Process event...
cache.add(event_id)
return {'statusCode': 200}
The issue is that when I return early for duplicates, I’m not propagating the context to any downstream service, which is fine, but I’m also not logging the deduplication in a way that ties back to the original trace. I want to see in Jaeger that an event was skipped, not just that it vanished.
Is there a standard pattern for this? Should I be using a distributed cache like Redis with TTLs for the dedup keys? And how do I inject the context into a log statement or a span attribute for the skipped events without creating a new span?
I’m running this on AWS Lambda, so cold starts are also a factor. Any pointers on the best way to handle this with the OTel SDK?