Implementing Real-Time Translation Overlays for Cross-Language Agent-Customer Conversations

What This Guide Covers

This guide details the architectural implementation of real-time translation overlays for Genesys Cloud CX. You will configure the Speech-to-Text (STT) and Text-to-Speech (TTS) engines to handle distinct language pairs, wire them into Architect flows, and deploy the Translation Overlay within the Agent Desktop. The end result is a seamless experience where an English-speaking agent receives live transcriptions and synthesized audio in their native language while the customer speaks Spanish, and vice versa.

Prerequisites, Roles & Licensing

• Licensing:
  • Genesys Cloud CX 3 license for all agents and supervisors involved.
  • Speech Analytics add-on is mandatory for the underlying STT/TTS infrastructure.
  • Translation feature flag must be enabled by Genesys Support. This is not available by default on all orgs.
• Permissions:
  • Speech Analytics > Manage Speech Analytics
  • Architect > Edit Flows
  • Routing > Edit Queues
  • Administration > Manage Settings
• External Dependencies:
  • No external APIs are required. The translation logic resides within the Genesys Cloud Speech Services layer.
  • Ensure your organization has configured at least two distinct STT engines (e.g., one for English-US, one for Spanish-ES).

The Implementation Deep-Dive

1. Configuring Polyglot Speech Engines

The foundation of real-time translation is the decoupling of the input language from the output language. Genesys Cloud does not perform translation within a single monolithic engine. Instead, it uses a pipeline: Source Language STT → Translation Service → Target Language TTS (optional, for outbound audio) or Target Language Text (for overlay).

You must configure two separate Speech-to-Text engines. One engine handles the customer’s language, and the other handles the agent’s language. While Genesys supports multi-language models, explicit separation provides better accuracy control and latency management.

Step 1.1: Create the Source Language STT Engine (Customer Side)
Navigate to Admin > Speech > Speech-to-Text. Create a new engine.

• Name: STT-Spanish-ES-RealTime
• Language: Spanish (Spain) or Spanish (Latin America) depending on your demographic.
• Model: Select the Real-Time model. Do not select Batch or Archive. Batch models introduce latency unacceptable for live conversation.
• Punctuation & Diarization: Enable both. Diarization is critical for the overlay to distinguish between the agent and the customer.

Step 1.2: Create the Target Language STT Engine (Agent Side)
Create a second engine for the agent’s native language.

• Name: STT-English-US-RealTime
• Language: English (United States)
• Model: Real-Time
• Punctuation & Diarization: Enable both.

The Trap: Language Mismatch in Hybrid Conversations
A common misconfiguration occurs when an agent speaks English but the STT engine assigned to the interaction is set to Spanish because the customer started the call in Spanish. If you assign a single STT engine to the entire interaction, the engine will attempt to transcribe the agent’s English audio using a Spanish language model. This results in gibberish transcripts or complete silence.

The Solution: You must dynamically switch the STT engine mid-flow or use a dual-engine approach. The recommended architecture is to use the Translation Overlay feature, which abstracts this complexity. However, the overlay relies on the underlying engines being correctly provisioned. Ensure you have distinct engines for every language pair you support.

2. Architect Flow Integration for Translation

Architect must be configured to initiate the translation session. The translation overlay is not a passive background process; it is an active media stream manipulation. You must inject the translation logic into the media path.

Step 2.1: The Set Translation Action
In your Architect flow, after the customer has been identified and the language preference is known (via IVR selection or historical data), you must use the Set Translation action.

{
  "id": "set-translation-action",
  "type": "setTranslation",
  "properties": {
    "from": "es-ES",
    "to": "en-US",
    "mode": "full-duplex"
  },
  "transitions": [
    {
      "label": "Success",
      "entityId": "queue-connect-action",
      "entityName": "Queue Connect",
      "type": "action"
    }
  ]
}

Key Parameters:

• from: The ISO code of the customer’s language.
• to: The ISO code of the agent’s language.
• mode: Use full-duplex for voice calls. This allows simultaneous translation of both parties. Use half-duplex only for specific compliance scenarios where you require sequential speaking turns.

The Trap: Setting Translation Too Late
If you place the Set Translation action after the Queue Connect action, the translation engine will not attach to the media stream. The initial seconds of the conversation will be untranscribed. Always place Set Translation immediately before the Queue Connect or Connect action.

Step 2.2: Handling Language Detection Failures
If you do not know the customer’s language a priori, you cannot use the Set Translation action directly. You must use the Detect Language action first.

{
  "id": "detect-lang-action",
  "type": "detectLanguage",
  "properties": {
    "timeout": 5000,
    "minConfidence": 0.85
  },
  "transitions": [
    {
      "label": "Detected",
      "entityId": "set-translation-action",
      "entityName": "Set Translation",
      "type": "action"
    },
    {
      "label": "Not Detected",
      "entityId": "fallback-english-queue",
      "entityName": "Fallback Queue",
      "type": "action"
    }
  ]
}

Architectural Reasoning: The minConfidence threshold is critical. Setting it too low (e.g., 0.5) causes the system to guess the language, leading to incorrect translation engines being loaded. Setting it too high (e.g., 0.99) causes frequent fallbacks. A value of 0.85 is the industry standard for real-time telephony.

3. Agent Desktop Configuration and Overlay Deployment

The translation overlay is a UI component within the Genesys Cloud Agent Desktop. It displays the live transcript in the agent’s native language.

Step 3.1: Enable Translation Overlay in Settings
Navigate to Admin > Settings > Agent Settings.

• Locate Translation Overlay.
• Set Enabled to true.
• Configure Display Mode:
  • Inline: Translations appear within the main transcript window.
  • Side-by-Side: Original transcript on the left, translated transcript on the right. This is recommended for training and quality assurance.

Step 3.2: Queue-Specific Translation Rules
Translation can be enforced at the queue level. Navigate to Admin > Routing > Queues.

• Select the target queue.
• In the Translation section, select Enforce Translation.
• Set Source Language to Auto-Detect or a specific language.
• Set Target Language to Agent's Preferred Language.

The Trap: Agent Preference Override
If you enforce translation at the queue level, it overrides the agent’s individual preference. However, if an agent has their desktop language set to English but the queue enforces Spanish as the target, the overlay will display Spanish text. This is counter-intuitive. Always ensure the Target Language in the queue settings matches the dominant language of the agent pool.

Step 3.3: Audio Translation (TTS Injection)
For voice calls, you can inject synthesized audio. This is distinct from the text overlay.

• In the Set Translation action in Architect, add the tts parameter.

"properties": {
  "from": "es-ES",
  "to": "en-US",
  "mode": "full-duplex",
  "tts": {
    "enabled": true,
    "voice": "en-US-JennyNeural"
  }
}

This causes the system to play back the translated audio to the agent via their headset. The customer hears the agent’s original audio. The agent hears the TTS-generated translation of the customer’s speech.

Architectural Reasoning: TTS injection introduces latency. The translation pipeline is: Audio → STT → Text → Translate → TTS → Audio. This can add 500ms to 2s of delay. For real-time conversation, this can cause talk-over issues. Use TTS injection only if the latency is acceptable for your use case. Otherwise, rely on the text overlay.

4. Validation and Monitoring

Real-time translation is invisible to the end-user until it fails. You must implement robust monitoring.

Step 4.1: Transcript Review
Navigate to Analytics > Speech Analytics > Transcripts.

• Filter by Translation: Enabled.
• Review the Source Text and Translated Text columns.
• Verify that the diarization tags (Speaker 1, Speaker 2) correctly map to Customer and Agent.

Step 4.2: Latency Monitoring
Use the Genesys Cloud API to monitor translation latency.

GET /api/v2/analytics/evaluations/conversations?entityId={conversationId}

Look for the translationLatency metric. If this exceeds 1.5 seconds, agents will experience cognitive load.

The Trap: Ignoring Confidence Scores
The translation engine returns a confidence score for each segment. Low confidence scores indicate potential translation errors. Do not ignore these. Configure a rule in Speech Analytics to flag transcripts where the average translation confidence falls below 0.7. These calls should be prioritized for QA review.

Validation, Edge Cases & Troubleshooting

Edge Case 1: Code-Switching (Code-Mixing)

The Failure Condition:
A customer speaks primarily Spanish but inserts English terms (e.g., “I need to reset my password”). The translation engine may fail to translate the English segment, leaving it as-is in the target transcript, or it may translate the English word incorrectly into Spanish.

The Root Cause:
Standard STT engines are monolingual. They do not handle code-switching well. The STT engine for Spanish may misinterpret the English word as a Spanish homophone or ignore it.

The Solution:
Use a Multilingual STT Engine if available in your region. Genesys Cloud supports some multilingual models that can detect and transcribe code-switched content. If a multilingual model is not available, you must rely on the Detect Language action at a segment level, which is complex to implement in Architect. A simpler workaround is to train agents to ask for clarification when the transcript shows unexpected English words in a Spanish conversation.

Edge Case 2: Accents and Dialects

The Failure Condition:
A customer speaks with a strong regional accent (e.g., Andalusian Spanish). The STT engine configured for Spanish (Spain) may struggle, resulting in low confidence transcripts and poor translation.

The Root Cause:
STT models are trained on specific dialects. Spanish (Spain) differs significantly from Spanish (Mexico) in pronunciation and vocabulary.

The Solution:
Deploy multiple STT engines for the same language but different regions. Use the Detect Language action with a broader confidence threshold to identify the region, then route to the appropriate Set Translation action. Alternatively, use the es-ES engine for all Spanish calls and accept a slight drop in accuracy, as the translation layer can often compensate for minor transcription errors.

Edge Case 3: Latency Spike During Peak Load

The Failure Condition:
During peak call volume, the translation overlay stops updating or shows delayed text.

The Root Cause:
The Speech Analytics service has a throughput limit. If the number of concurrent translated conversations exceeds the licensed capacity or the regional service limit, the queue for STT/TTS processing grows.

The Solution:
Monitor the Speech Analytics Utilization dashboard in Admin. If utilization approaches 100%, you must scale your license count or implement a fallback mechanism. In Architect, add a Try/Catch block around the Set Translation action. If the translation service returns a timeout error, fall back to a standard queue without translation.

{
  "id": "try-catch-translation",
  "type": "tryCatch",
  "properties": {
    "retryCount": 0
  },
  "transitions": [
    {
      "label": "Try",
      "entityId": "set-translation-action",
      "entityName": "Set Translation",
      "type": "action"
    },
    {
      "label": "Catch",
      "entityId": "queue-no-translation",
      "entityName": "Queue No Translation",
      "type": "action"
    }
  ]
}

Official References

• Genesys Cloud Translation Documentation
• Architect Set Translation Action
• Speech Analytics Real-Time Engine Configuration
• Agent Desktop Translation Overlay Settings