# Configuring Multi-Region Redundancy for Genesys Cloud Speech AI Models by Replicating Custom Language Model N-gram Databases Across AWS Regions Using Cross-Region Replication Rules
## What This Guide Covers
This guide details configuring cross-region replication of Genesys Cloud Speech AI custom language model N-gram databases in AWS. Successfully completing this configuration establishes redundancy, reducing downtime during regional outages and ensuring consistent model performance regardless of call routing location. The final result is a replicated N-gram database accessible by Speech AI models in multiple AWS regions.
## Prerequisites, Roles & Licensing
- **Genesys Cloud CX:** Requires Genesys Cloud CX with Speech AI enabled.
- **Licensing:** Speech AI licensing (pay-as-you-go or committed seats). WEM add-on is not required for replication itself, but is crucial for monitoring replication status.
- **AWS Account Access:** Access to the AWS console with permissions to manage S3 buckets and IAM roles in *both* the primary and secondary AWS regions.
- **IAM Permissions (Genesys Cloud User):** The user performing the configuration must have the following permissions within Genesys Cloud:
- `Speech Integration > Speech AI > Language Model > View`
- `Speech Integration > Speech AI > Language Model > Edit`
- `Speech Integration > Speech AI > Language Model > Delete` (required for initial setup, but can be revoked after replication rules are created).
- **AWS IAM Role Permissions:** An IAM role within AWS that Genesys Cloud will assume, with the following permissions:
- `s3:ListBucket` on both source and destination S3 buckets.
- `s3:GetObject` on the source S3 bucket.
- `s3:PutObject` on the destination S3 bucket.
- `s3:DeleteObject` on the destination S3 bucket.
- `s3:ReplicateObject` on the source S3 bucket.
- **AWS Regions:** Two distinct AWS regions are required - a primary region where Speech AI processing currently occurs, and a secondary region for disaster recovery.
- **N-gram Database:** Existing custom language model N-gram databases in Genesys Cloud Speech AI.
## The Implementation Deep-Dive
### 1. Identifying and Preparing the S3 Bucket
Genesys Cloud Speech AI stores custom language model N-gram databases as S3 objects within AWS. The first step is to identify the S3 bucket associated with your Genesys Cloud account. This bucket is *not* directly visible within the Genesys Cloud UI; it requires AWS console access.
1. **Locate the S3 Bucket:** Within the AWS console, navigate to S3. Search for buckets whose names include `genesyscloud-speechai`. There will likely be multiple buckets; identify the one containing the N-gram databases you wish to replicate. These will be identifiable by their object key structure containing the `ngram` prefix.
2. **Enable Cross-Region Replication:** On the identified source S3 bucket, navigate to the “Management” tab and then “Replication rules”. Create a new replication rule.
3. **Configure Replication Rule:**
- **Destination Region:** Select the secondary AWS region.
- **Destination Bucket:** Select or create a new S3 bucket in the secondary region. *Crucially*, the bucket name must be different from the source bucket name.
- **Replication Scope:** Choose to replicate all objects, or filter based on object key prefix (e.g., `ngram/`).
- **Storage Class:** Select the appropriate storage class for the replicated data.
- **Encryption:** Configure encryption as needed. Ensure the encryption configuration is consistent between the source and destination buckets.
4. **IAM Role:** Select or create an IAM role that allows S3 to replicate objects. This role requires the permissions listed in the Prerequisites section.
**The Trap:** Failing to use a *different* bucket name in the destination region. S3 prevents replicating to a bucket with the same name, resulting in a silent failure of the replication rule. This is a common oversight and leads to a false sense of security.
### 2. Linking AWS IAM Role to Genesys Cloud
Genesys Cloud requires an IAM role to access the S3 buckets on your behalf. This is done within the Speech AI settings in Genesys Cloud.
1. **Navigate to Speech AI:** In Genesys Cloud, go to Admin > Speech & Media > Speech AI.
2. **AWS Credentials:** Select “AWS Credentials.”
3. **Add New Credentials:** Enter a descriptive name for the credentials.
4. **IAM Role ARN:** Enter the ARN (Amazon Resource Name) of the IAM role you created in AWS. This role *must* have the permissions outlined in the Prerequisites section.
5. **Region:** Select the primary AWS region.
6. **Save:** Save the AWS credentials. Genesys Cloud will attempt to validate the IAM role.
**The Trap:** Providing an IAM role ARN that does not have the correct permissions. Genesys Cloud will *not* provide detailed error messages if the IAM role is missing permissions; the replication simply won't work. Validate the IAM role meticulously.
### 3. Configuring Replication Schedule & Verification
After setting up the S3 replication rule and linking the IAM role, it's time to verify that the replication is working. S3 replication is asynchronous, so it will not be immediate.
1. **Monitor Replication:** Within the AWS console, monitor the S3 bucket replication status. You can view the replication progress in the “Replication” tab of the source S3 bucket. Look for any errors or failed replications.
2. **Initial Synchronization:** The initial synchronization may take a significant amount of time, depending on the size of the N-gram databases.
3. **Verification:** Once replication is complete, verify the replicated data in the destination S3 bucket. Ensure that all N-gram database files have been successfully copied. Compare object counts and file sizes between the source and destination.
**The Trap:** Assuming immediate replication. S3 replication is event-driven and asynchronous. Do not proceed to testing until the initial synchronization is complete.
### 4. Failing over Speech AI Models (Disaster Recovery Testing)
While the replication is active, the secondary region is in a passive state. To validate the redundancy, a failover test is crucial.
1. **Update AWS Credentials in Genesys Cloud:** Update the Speech AI AWS credentials to point to the secondary AWS region.
2. **Test Speech AI Models:** Deploy a test call flow using Speech AI models. Verify that the models function as expected, processing audio from the secondary region.
3. **Monitor Performance:** Monitor the performance of the models in the secondary region. Ensure that latency and accuracy are comparable to the primary region.
4. **Failback:** After testing, revert the AWS credentials back to the primary region.
**The Trap:** Failing to test the failover procedure thoroughly. A successful replication setup does not guarantee a successful failover. Regularly test the failover process to ensure that it works as expected.
## Validation, Edge Cases & Troubleshooting
### Edge Case 1: Replication Lag
- **The Failure Condition:** N-gram databases are not updated in the secondary region in a timely manner, resulting in stale models during a failover.
- **The Root Cause:** Replication lag due to network congestion, high object volume, or S3 throttling.
- **The Solution:** Monitor the replication lag using CloudWatch metrics. Consider increasing the replication rate or using S3 Transfer Acceleration to reduce lag. Optimize the N-gram database update frequency to minimize the impact of replication lag.
### Edge Case 2: IAM Role Expiration
- **The Failure Condition:** The IAM role used by Genesys Cloud expires, causing Speech AI to lose access to the S3 buckets.
- **The Root Cause:** The IAM role has a limited validity period.
- **The Solution:** Configure the IAM role with a long validity period or implement a rotation policy to automatically renew the role before it expires. Monitor the role's expiration date and renew it proactively.
### Edge Case 3: Replication Rule Errors
- **The Failure Condition:** The replication rule encounters errors, preventing the N-gram databases from being replicated.
- **The Root Cause:** Incorrect IAM permissions, network connectivity issues, or storage class incompatibilities.
- **The Solution:** Review the S3 replication rule logs for error messages. Verify that the IAM role has the correct permissions and that there are no network connectivity issues. Ensure that the storage classes are compatible between the source and destination buckets.
## Official References
- [Genesys Cloud Resource Center - Speech AI](https://help.mypurecloud.com/hc/en-us/articles/360061802051-Speech-AI)
- [Genesys Cloud Developer Center - Speech AI API](https://developer.genesys.cloud/reference/speech-ai)
- [AWS S3 Cross-Region Replication Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html)
- [AWS IAM Best Practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html)