Data Action OTel Span Leak - High Latency on /data-actions/v2/actions/{actionId

Architect & Flows
1

We’re seeing…odd behavior…with Data Action execution times…specifically with spans leaking. It’s a performance hit-latency jumped from 80ms to 3s on a simple webhook call to /data-actions/v2/actions/a1b2c3d4-e5f6-7890-1234-567890abcdef.

The OTel SDK version is 1.17.0…and context propagation should be working…it’s injecting the trace ID and span ID headers…right? But Jaeger shows multiple root spans originating from the same interaction handle-it looks like the Data Action is re-initializing the tracer on every retry, which isn’t ideal. Shouldn’t it just pick up the existing context? Memory overhead is also concerning-multiple span contexts per interaction are adding up… quickly.

The Architect flow is basic-a single Data Action node. No loops…no branching…just a straight call to the action. Is the retry logic resetting the context…again?

2

Thanks, - that’s a tricky one.

Seeing span leaks in Data Actions is annoying. Here’s what helped us:

  1. Check the tracePropagation config in your Data Action. It needs to be set to true.
  2. Ensure your webhook handler is correctly propagating the headers - X-Genesys-Trace-ID and X-Genesys-Span-ID.
  3. Repro in a lower env first - prod perf is fragile.

Anyone seen this?

1 Like
3

that trace propagation stuff can be a pain, yeah. we’ve hit similar slowdowns with Data Actions before - it’s almost always something wonky with the headers.

Cause: It sounds like the spans aren’t stitching together properly. The root cause is usually the Data Action isn’t passing the trace context to the webhook, or your handler isn’t picking it up. That’s what the earlier post’s pointing at with the tracePropagation setting and header check.

Solution: Double-check the payload config for the Data Action. You’ll want to make sure you’re including the X-Genesys-Trace-ID and X-Genesys-Span-ID in the outbound headers. Here’s what it looks like in JSON -

{
 "tracePropagation": true,
 "outboundHeaders": [
 {
 "name": "X-Genesys-Trace-ID",
 "value": "{{Conversation.traceId}}"
 },
 {
 "name": "X-Genesys-Span-ID",
 "value": "{{Conversation.spanId}}"
 }
 ]
}

But… are you sure you’re actually using the trace ID and span ID in your webhook handler? Sometimes the handler code forgets to forward them on to downstream services and that breaks the chain. Also, is the webhook handler even handling the headers? Just wondering if there’s something simple going on there.

1 Like