Analytics Alert - Event to PagerDuty - Duplicate Incidents

Fun one today. Think of the Genesys Cloud Analytics API as a leaky faucet - it drips events, and we’re trying to catch every drop with PagerDuty. The issue isn’t missing events, it’s getting the same event triggering multiple PagerDuty incidents, almost immediately after each other. It’s like trying to fill a bucket with that leaky faucet, and the overflow protection is way too sensitive.

We’ve got an Analytics alert set up monitoring queue abandon rate. When it crosses 5%, it triggers a webhook to our internal service, which then formats a PagerDuty v2 event. The alert itself seems accurate - GC Analytics shows the rate jumping, and the webhook fires. The problem is PagerDuty shows two, sometimes three, incidents for the same alert evaluation.

The service is written in Node.js, using the PagerDuty Events API v2 SDK (version 2.4.3). Here’s the relevant code snippet for creating the event:

const pagerDuty = require('pagerduty-events-api');
const client = new pagerDuty.Client();

async function createPagerDutyIncident(abandonRate, queueName) {
 const event = {
 routing_key: 'YOUR_ROUTING_KEY',
 event_action: 'trigger',
 payload: {
 summary: `High Abandon Rate - ${queueName}`,
 detail: `Abandon rate exceeded 5% (${abandonRate}%)`,
 source: 'Genesys Cloud Analytics'
 },
 dedup_key: 'abandon-rate-' + queueName //trying to dedup on queue name
 };

 try {
 await client.trigger(event);
 console.log('PagerDuty incident created.');
 } catch (error) {
 console.error('Error creating PagerDuty incident:', error);
 }
}

We’ve tried setting dedup_key - initially, just a static value, then the queue name. Doesn’t help. The PagerDuty UI shows no correlation ID - it’s as if the requests are just arriving independently. Network traces show the webhook firing, and the POST requests to PagerDuty are successful (202 responses). I also checked the GC alert history - it’s only triggering once per rate crossing, so it’s not a double-fire from the source. The GC environment is prod, using the Miami region. The webhook endpoint is TLS 1.2 secured.

It feels like a race condition somewhere, or maybe PagerDuty is having issues with handling similar events rapidly. The mic stays hot, just need to stop the flood.

Hey everyone,

It sounds like the event deduplication is not working as you expect. The documentation for the Analytics API states that “event data is sent on a best-effort basis, and duplicate events may occur”. It’s not ideal, but it explains why you’re getting the duplicates.

Here’s a few things you can check:

• The PureCloudPlatformClientV2 doesn’t have a built-in deduplication feature for Analytics events, so you’ll need to handle it in your PagerDuty integration logic. You’ll need to store event IDs, and ignore duplicates within a short time window.
• Check the alert condition definition - maybe the alert is too sensitive and triggers on slight fluctuations.
• Consider increasing the threshold for the alert. 5% might be too low, and any minor fluctuation will trigger it.
• You can use the metrics.aggregation.interval parameter when creating the alert to reduce the frequency of event triggers. The documentation says the minimum is 60 seconds.

It’s a bit of work to add the deduplication logic, but it should solve the duplicate incident problem.