Recording Exports - Lambda Data Action Timeout

Genesys Cloud WFM
1

The screen recording export to S3 is failing again - different error this time, but same EventBridge setup. Rule is firing on interaction.screen.recording.created, routing to a Lambda Data Action - PUT object. It’s timing out, and the Lambda isn’t even being invoked half the time. Thought we fixed this last week.

2024-02-29T15:32:58.789Z 64a1b2c3-d4e5-6789-0123-456789abcdef ERROR DataActionExecutor - Lambda execution failed with error: TimeoutError: Lambda function execution exceeded maximum timeout of 30 seconds.

We’ve increased the Lambda timeout to 60 seconds in the CloudFormation stack - it didn’t help. Something’s blocking the initial EventBridge trigger, maybe? This whole thing is a mess, and the Architect flow isn’t showing any errors. The recording ID is definitely uppercase in the event.

1 Like
2

Cause:

The intermittent Lambda invocation suggests the EventBridge rule isn’t consistently triggering, or there’s an issue with the permissions configuration. A timeout error within the Lambda itself points to processing delays - likely related to the recording data size or the S3 PUT operation. We’ve encountered similar latency with larger recording files.

Solution:

First, verify the EventBridge rule’s target configuration. Ensure the Lambda function’s permission policy allows EventBridge to invoke it. Check the CloudWatch logs for EventBridge - it should indicate if the rule is failing to match events, or if there are permission errors during invocation.

Next, optimize the Lambda function itself. Consider these points:

  • Asynchronous S3 PUT: Use the putObject method with a callback to handle the S3 operation asynchronously. This prevents the Lambda from blocking while waiting for the PUT to complete.
const AWS = require('aws-sdk');
const s3 = new AWS.S3();

exports.handler = async (event) => {
 const bucketName = 'your-s3-bucket-name';
 const key = 'recording-' + Date.now() + '.mp4'; // Or whatever naming convention
 const body = event.detail.recordingUri; // Adjust as needed based on EventBridge event structure

 const params = {
 Bucket: bucketName,
 Key: key,
 Body: body,
 };

 return new Promise((resolve, reject) => {
 s3.putObject(params, (err, data) => {
 if (err) {
 console.error('Error uploading to S3:', err);
 reject(err);
 } else {
 console.log('Successfully uploaded to S3:', data);
 resolve();
 }
 });
 });
};

  • Increase Lambda Timeout: Increase the Lambda function’s timeout duration. Start with 60 seconds and adjust based on observed processing times.

  • Memory Allocation: Increase the Lambda’s memory allocation. More memory can improve performance for data-intensive operations.

  • Error Handling: Implement solid error handling and retry logic within the Lambda. This can help handle transient S3 errors or network issues.

Regarding the inconsistent invocation - could you confirm the EventBridge rule pattern matches the event details accurately? We’ve seen issues when the event structure changes slightly, causing the rule to miss events. Are you filtering by any specific recording metadata within the EventBridge rule?

3

That timeout’s almost certainly the Lambda’s execution time hitting the limit- it’s a recurring pattern with the recording exports, and EventBridge isn’t going to give it more time just because it failed to invoke consistently at first. Try increasing the Lambda’s timeout to 60 seconds, then check the invocation logs to see if that solves the immediate problem- sometimes it’s that simple. If it’s still timing out, you need to optimize the S3 PUT operation inside the Lambda- a direct boto3.client('s3').put_object() call with multipart upload can help with larger files. Here’s a snippet of what that looks like, assuming you’ve already configured the S3 client with the correct credentials and bucket name: python import boto3 s3 = boto3.client('s3') def upload_recording(bucket_name, key, file_data): try: s3.upload_fileobj(file_data, bucket_name, key, {'ContentType': 'audio/mpeg'}) return True except Exception as e: print(e) return False The ContentType is important- double check that it matches the actual recording format. Also check the Lambda’s memory allocation- a low memory setting can artificially increase execution time.

1 Like