Hey folks,
I’m hitting a wall with our Genesys Cloud Webhook integration for WEM adherence events. We have a simple Python Flask app listening on an AWS ECS container to process routing.queue.member.adherence events. It works fine during low volume, but as soon as the morning rush hits, my container scales up and occasionally throws a 503 Service Unavailable if the health check fails or if the request queue backs up.
Genesys Cloud retries the webhook, but it seems to spam my endpoint until it gives up, and we lose those adherence records. I need to implement a Dead Letter Queue (DLQ) pattern so failed events go to SQS for retry later instead of just disappearing.
Here is the current endpoint code:
@app.route('/webhook/wem', methods=['POST'])
def handle_wem_event():
try:
payload = request.get_json()
# Process adherence logic
process_adherence(payload)
return '', 200 # Genesys expects 2xx
except Exception as e:
log.error(f"Failed processing: {e}")
return '', 500 # This triggers the retry storm
The issue is that when I return 500, Genesys retries immediately. I want to return a 200 to acknowledge receipt, but then asynchronously push the payload to an SQS DLQ if the processing fails.
I tried this:
except Exception as e:
sqs.send_message(QueueUrl=DLQ_URL, MessageBody=json.dumps(payload))
return '', 200
But sometimes the SQS call itself times out, causing a 500 again. Is there a standard way to handle this? I’m using boto3 for SQS. Should I be using a background thread here? The docs mention webhook reliability but don’t really cover DLQ implementation for consumers.
Any code examples for a solid retry pattern in Python that plays nice with Genesys Cloud’s retry logic?