Event Hub webhook duplicate delivery on AGENT_STATE_CHANGE

NICE CXone API & Integrations
1 
app.post('/webhooks/cxone', (req, res) => {
 const { eventId, eventType, payload } = req.body;
 
 if (eventType === 'AGENT_STATE_CHANGE') {
 console.log(`Processing state change for ${payload.agentId}: ${payload.newState}`);
 // Logic to sync state to Salesforce
 }
 
 res.status(200).send('OK');
});

The middleware is receiving the EXACT same eventId three or four times for a single state transition. It’s causing a race condition in the Salesforce sync because the updates are hitting the API out of order.

  • Node.js v18.16.0
  • NICE CXone Event Hub subscription
  • Webhook endpoint returning 200 OK within 200ms
  • Occurs mostly during peak morning hours in IST

The logs show the first request is processed and acknowledged, but then two more identical payloads arrive seconds later. This is REALLY messing up the agent state tracking.

anyone seen this?

2

Event Hub SDK. Hub retries hit if the 200 OK isn’t processed fast enough by the load balancer. Just cache the eventId in Redis for 60s and drop duplicates.

if (await redis.get(event.eventId)) return res.sendStatus(200);
await redis.set(event.eventId, '1', 'EX', 60);
2 Likes
3 
const isDuplicate = await redis.set(eventId, '1', 'NX', 'EX', 60);
if (!isDuplicate) return res.sendStatus(200);

The duplicate delivery is akin to a TCP retransmission where the acknowledgment is delayed, prompting the sender to resend the segment despite the original being in flight. This aligns with the idempotency requirements outlined in Request for Comments 7231.

Error State Root Cause Resolution
Duplicate eventId Network latency exceeding timeout Implement distributed locking
2 Likes
4

The Redis cache idea in the earlier reply is spot on. We saw this with Event Hub when the network lagged during peak shifts. If the 200 OK doesn’t hit the hub fast enough, it just fires again. Is the middleware using a load balancer or a proxy that might be delaying the ack?