I have a C# ASP.NET Core backend service that consumes webhooks from Genesys Cloud. The endpoint is /api/webhooks/conversations. Everything works fine when the load is normal. I get the JSON payload, process it, and return 200 OK within the timeout window. The issue arises during peak hours or when my internal database is slow. The Genesys webhook server gets a 500 Internal Server Error because my app throws an unhandled exception or times out. Genesys retries the delivery, but if my service is still down, those events are lost. I need to implement a Dead Letter Queue (DLQ) pattern to catch these failures and retry them later, or at least store them for manual review.
Here is my current webhook controller method:
[HttpPost("conversations")]
public async Task<IActionResult> HandleWebhook([FromBody] ConversationEvent payload)
{
try
{
await _conversationService.ProcessEvent(payload);
return Ok();
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to process webhook");
// Currently returning 500 which triggers Genesys retry
return StatusCode(500, "Processing failed");
}
}
The problem is that returning 500 tells Genesys to retry immediately. If my DB is down, the retries just pile up and fail again. I want to return 200 OK to acknowledge receipt, then move the payload to a queue (like Azure Service Bus or RabbitMQ) for async processing. If the async processing fails, I want to move it to a DLQ.
My question is: how do I structure this in C#? Should I use a background service with IHostedService? Or is there a better way to handle this within the controller scope? I also need to make sure I don’t drop any events. If I return 200 and then crash before pushing to the queue, the event is gone. I need a reliable way to buffer the request.
Also, Genesys docs say they retry for a certain period. What is the exact retry window? If I implement a DLQ, do I need to worry about duplicate events if Genesys retries while I am processing the original request? I am using .NET 6 and Azure SQL. Any code examples for this pattern would be great. I have looked at some samples but they are mostly for simple logging. I need something production-ready.