We are building a custom analytics pipeline that ingests real-time notification events from Genesys Cloud via WebSocket and forwards them to a Kafka topic. The goal is to capture conversation state changes and wrap-up codes for downstream APM correlation in New Relic.
The connection to the endpoint wss://api.mypurecloud.com/api/v2/analytics/events establishes successfully with a 101 Switching Protocols response. However, under moderate load (approx. 50 events/sec), the consumer starts dropping messages. The Kafka producer isn’t reporting errors, which suggests the data never reaches the processing logic.
Here is the relevant snippet handling the incoming frames:
ws.on('message', (data) => {
try {
const events = JSON.parse(data);
// events is an array of notification objects
events.forEach(event => {
producer.send({
topic: 'gc-notifications',
messages: [{ value: JSON.stringify(event) }]
}).catch(err => console.error('Kafka error:', err));
});
} catch (err) {
console.error('Parse error:', err);
}
});
The issue seems to be that the ws library buffer fills up faster than the asynchronous producer.send calls resolve. We’ve noticed that the process memory usage spikes, and eventually, the WebSocket connection closes with code 1006 (Abnormal Closure).
We tried adding a backpressure mechanism by checking producer.isConnected() and pausing the stream, but the GC WebSocket doesn’t seem to support standard flow control headers. We’re also seeing some latency in the New Relic custom events, which makes debugging the exact drop point difficult.
Is there a recommended pattern for handling high-volume Genesys notification streams in Node.js? Should we be batching the Kafka sends instead of firing per-event? We’ve considered using the @genesyscloud/platform-client-sdk but need the raw stream for specific filtering logic before Kafka ingestion.
Any pointers on managing the backpressure between the WebSocket stream and the Kafka producer would be appreciated. We’re currently stuck on this bottleneck.