We are implementing a real-time interaction processing pipeline where Genesys Cloud pushes interaction events via EventBridge to an AWS Lambda function. The goal is to update our internal CRM system immediately upon receiving these events, specifically for high-volume call center operations.
The Lambda function is configured with provisioned concurrency of 100 instances. We have set the maximum concurrent executions to 500. Despite this configuration, we are observing significant throttling during peak hours. The CloudWatch logs show a high number of Throttles metrics, and the EventBridge invocation fails with ResourceExhausted errors.
The Lambda function code is written in Python and uses the requests library to make HTTP POST calls to our internal CRM endpoint. The function is designed to be idempotent and handles retries internally.
Here is the relevant portion of the Lambda handler:
import json
import requests
import boto3
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
# Process the EventBridge event
for record in event['Records']:
body = json.loads(record['body'])
# Extract interaction details
interaction_id = body.get('interactionId')
# Update CRM
try:
response = requests.post(
'https://internal-crm.example.com/api/v1/interactions',
json=body,
headers={'Authorization': 'Bearer <token>'},
timeout=5
)
response.raise_for_status()
except requests.exceptions.RequestException as e:
logger.error(f'Failed to update CRM for interaction {interaction_id}: {e}')
# Retry logic here
pass
return {
'statusCode': 200,
'body': json.dumps('Successfully processed events')
}
We have verified that the internal CRM endpoint can handle the load by testing with a load generator. The issue seems to be on the AWS side. We have also checked the Lambda function’s memory and timeout settings, which are set to 512 MB and 10 seconds, respectively.
The EventBridge rule is configured to send events to the Lambda function with a batch size of 100. We have also enabled the MaximumRetryAttempts setting to 3.
We are looking for best practices to handle high-volume event processing from EventBridge without hitting Lambda concurrency limits. Are there any specific configurations or patterns we should consider? We have already tried increasing the provisioned concurrency, but the throttling persists.