Trying to understand the correct pattern for implementing a dead letter queue when consuming Genesys Cloud webhooks via EventBridge.

I have a Lambda function subscribed to an EventBridge rule that triggers on routing.queue.member.wrapped events. The Lambda processes the payload and updates an external CRM. When the CRM is down, the Lambda throws a 5xx error. Genesys Cloud retries the webhook delivery according to its internal schedule, but EventBridge also retries the Lambda invocation. This results in duplicate processing and eventual saturation of the retry limit. I want to move failed payloads to an SQS DLQ immediately upon the first 5xx response from my consumer, rather than relying on Genesys’s retry mechanism or Lambda’s default retry policy. The webhook registration uses the standard /api/v2/architect/webhooks endpoint. My current EventBridge rule target is the Lambda ARN. Should I be using Step Functions as a target to handle the error branching, or is there a way to configure the webhook payload to include a retry token that I can manage externally? I need to ensure idempotency while preventing the cascade of retries from both Genesys and AWS.

The easiest fix here is this is to decouple the webhook ingestion from the CRM update logic by using an SQS queue as a buffer. Direct Lambda invocations from EventBridge are synchronous in nature regarding the rule evaluation, and if the Lambda fails, EventBridge retries indefinitely unless configured otherwise, which compounds with Genesys Cloud’s own retry mechanism. This creates a thundering herd of duplicate events. Instead, configure your EventBridge rule to target an SQS queue. In your Lambda, use the AWS SDK for Go v2 to process messages from the queue. Implement a custom retry middleware using github.com/aws/aws-sdk-go-v2/service/sqs with exponential backoff. If the CRM update fails, do not throw a panic. Instead, send the failed message to a Dead Letter Queue (DLQ) associated with the main queue. Here is the Go structure for the DLQ target policy: {"DeadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:crm-dlq", "MaxReceiveCount": 3}. This ensures that transient CRM failures are retried locally without triggering Genesys Cloud retries. You can then monitor the DLQ for permanent failures. Use OpenTelemetry to trace the latency from webhook receipt to DLQ insertion. This pattern separates concerns: Genesys Cloud delivers to EventBridge, EventBridge buffers in SQS, and your Lambda handles business logic with controlled retries. Avoid mixing synchronous webhook retries with asynchronous queue retries. Ensure your Lambda handler returns a success response to SQS immediately after acknowledging receipt, even if the CRM update is pending in a background goroutine. This prevents SQS from redelivering the message prematurely.

The way I solve this is by ensuring the lambda returns a 200 immediately upon acknowledging the event, then offloading the heavy lifting. the suggestion above about sqs is solid, but don’t forget that eventbridge to sqs is asynchronous, so you lose the immediate feedback loop unless you handle retries in the consumer. in my laravel microservices, i use guzzle to fetch the batch from sqs. if the crm fails, i don’t let the lambda error out. instead, i push the failed message to an sqs dead letter queue with a retry count header. here is the basic php logic for the consumer:

$sqs = new SqsClient([...]);
$dlq = new SqsClient([...]);

// after processing fails
$dlq->sendMessage([
 'QueueUrl' => config('services.genesis.dlq_url'),
 'MessageBody' => $message->getBody(),
 'MessageAttributes' => [
 'retryCount' => ['DataType' => 'Number', 'StringValue' => (string)($count + 1)]
 ]
]);

this keeps the eventbridge rule happy and isolates the flaky crm calls.

The problem is that local mock servers often fail to replicate the exact retry jitter Genesys Cloud applies, leading to false confidence in your DLQ logic.

1. Configure your docker-compose.yml mock service to enforce strict backoff matching Genesys Cloud’s exponential decay.
2. Validate the X-Genesys-Cloud-Retry-Count header in your local test harness before deploying.

services:
 mock-gc:
 environment:
 - RETRY_POLICY=exponential
 - INITIAL_DELAY_MS=1000

You need to stop letting EventBridge handle the retry logic and handle it in the webhook handler itself. The suggestion about SQS is fine for decoupling, but it doesn’t stop Genesys Cloud from hammering your endpoint if you return 5xx. In PowerShell, I wrap my API calls in a try-catch block that explicitly returns a 200 OK to Genesys Cloud immediately upon receipt, regardless of downstream CRM status. If the CRM call fails, I log the failure to a local JSON file or a secondary API endpoint for later processing. This prevents the thundering herd effect described above.

powershell

Pseudo-code for webhook handler logic

try {

Acknowledge receipt immediately

Return-Response -StatusCode 200 -Body ‘{“status”: “received”}’

Process in background thread or separate function

Update-CrmRecord -Payload $Body
}
catch {

Log error but do not throw 5xx

Write-Log -Message “CRM Update Failed: $_” -Level Error

Still return 200 to stop GC retries

Return-Response -StatusCode 200 -Body ‘{“status”: “received”}’
}

This ensures Genesys Cloud stops retrying. You handle the idempotency in your CRM update logic.