Designing Agent Copilot Confidence Thresholds for Auto-Accept vs Manual Review Suggestions

How-To-Articles
1

Designing Agent Copilot Confidence Thresholds for Auto-Accept vs Manual Review Suggestions

What This Guide Covers

This guide details the architectural configuration of Genesys Cloud CX Agent Copilot to establish precise confidence thresholds for automatic action insertion versus manual review suggestions. The end result is a deterministic workflow where high-confidence, low-risk AI actions (such as standard data lookups or routine script confirmations) populate the agent workspace automatically, while lower-confidence or high-impact actions (such as financial transactions or complex data modifications) require explicit agent validation.

Prerequisites, Roles & Licensing

  • Licensing: Genesys Cloud CX 3 license with Agent Copilot add-on enabled.
  • Permissions:
    • Conversation > Conversation > Read
    • Architect > Architect > Read (for Flow configuration)
    • Administration > User > Edit (to assign Copilot features to users)
    • AI > Agent Copilot > Edit (to manage Copilot settings)
  • External Dependencies:
    • An active Conversation Intelligence deployment (for transcript ingestion).
    • A configured Task Management or Queue structure to handle manual review tasks if manual review is routed as a work item.
    • Access to the Genesys Cloud Developer Portal for custom API endpoints if integrating with external systems for validation.

The Implementation Deep-Dive

1. Configuring the Copilot Skill Set and Model Parameters

The foundation of any Copilot implementation is the definition of what the AI is allowed to do. In Genesys Cloud, this is not a single global toggle but a composite of skill definitions, prompt engineering constraints, and model temperature settings.

Step 1: Define the Skill Scope
Navigate to Admin > AI > Agent Copilot > Skills. You must create distinct skills for different operational domains. Do not create a single “General Support” skill. Instead, create:

  • Data_Lookup_Read_Only: Restricted to GET operations on CRM or database systems.
  • Transaction_Write_Risky: Restricted to POST/PUT operations involving financial or PII changes.
  • Script_Assistance: Restricted to text generation based on internal knowledge bases.

Step 2: Configure Model Parameters
In the Copilot Settings, adjust the Temperature and Top_P values.

  • Temperature: Set to 0.0 for deterministic, factual tasks (e.g., retrieving an order number). Set to 0.3 for generative tasks (e.g., summarizing a call).
  • The Trap: Setting Temperature to 0.0 for all skills causes the model to fail on creative or ambiguous queries, returning errors or generic refusals. Conversely, setting it too high for data lookups introduces hallucination risks where the model invents data fields that do not exist in the schema.
  • Architectural Reasoning: We separate skills to allow granular control over risk. A high-temperature model is acceptable for drafting an email summary but catastrophic for executing a database update. By isolating skills, we apply strict deterministic parameters to read-only actions and slightly more flexible (but still constrained) parameters to generative tasks.

Step 3: Define the System Prompt Constraints
Within each Skill definition, configure the System Prompt. This prompt must explicitly define the output format and confidence expectations.

{
  "system_prompt": "You are a support assistant. When asked to look up data, return a JSON object with fields: 'action_type', 'confidence_score', 'data_payload'. If confidence is below 0.9, set 'requires_review' to true. Do not invent data. If data is missing, return null."
}

2. Implementing the Confidence Threshold Logic in Architect

Genesys Cloud does not have a native “If Confidence < X Then Manual Review” toggle in the Copilot UI. You must implement this logic using Architect to intercept Copilot responses before they are presented to the agent. This requires using the AI > Generate Content node or a Script Task that calls the Copilot API, followed by a conditional branch.

Step 1: The Copilot Invocation Node
In your Architect Flow, use the AI > Generate Content node. Configure it to call the specific Skill defined above.

  • Input: Map the conversation_transcript or specific user_query to the prompt.
  • Output: Map the response to a flow variable, e.g., ${copilot_response}.

Step 2: Parsing the Confidence Score
The Copilot response will include a metadata object containing the confidence_score. You must extract this value.

// Example JavaScript snippet for a Script Task in Architect
function parseConfidence(response) {
    try {
        const parsed = JSON.parse(response.body);
        return {
            score: parseFloat(parsed.confidence_score),
            requiresReview: parsed.requires_review === true,
            payload: parsed.data_payload
        };
    } catch (e) {
        return { score: 0.0, requiresReview: true, payload: null };
    }
}

Step 3: The Conditional Branch (The Threshold Gate)
Create a Condition node after the parsing step.

  • Condition: ${parsed_confidence.score} >= 0.95
  • True Path (Auto-Accept): Proceed to the UI Integration node or Data Update node. The action is executed silently or displayed as a confirmed suggestion.
  • False Path (Manual Review): Route to a Task Creation node or a UI Notification node that flags the suggestion for agent review.

The Trap: The most common misconfiguration is treating the confidence_score as a linear probability. Genesys Cloud Copilot confidence scores are often calibrated to the specific skill. A 0.9 score in Data_Lookup may be less reliable than a 0.85 score in Script_Assistance due to the determinism of the underlying data source.
Architectural Reasoning: We implement a dynamic threshold based on the action_type. Read-only actions can have a lower threshold (0.90) because the cost of error is low (agent can correct it). Write actions must have a higher threshold (0.98) or always require manual review regardless of score. This is implemented by adding a second condition: AND ${action_type} != 'WRITE'.

3. UI Integration and Agent Experience Design

The final layer is how the agent interacts with these thresholds. Genesys Cloud provides the Agent Workspace API and Custom Components to render Copilot suggestions.

Step 1: Defining the Suggestion Widget
Use the Agent Workspace API to register a custom component that listens for Copilot events. The component must distinguish between auto_accepted and review_required states.

// Pseudocode for Custom Component Registration
genesys.agentspace.registerComponent('copilot-suggestion', {
    onEvent: (event) => {
        if (event.type === 'COPILOT_SUGGESTION') {
            const { confidence, requiresReview } = event.data;
            
            if (requiresReview || confidence < 0.95) {
                // Render as a "Draft" or "Review" card
                renderReviewCard(event.data.payload, confidence);
            } else {
                // Render as an "Action" card with auto-apply
                renderAutoApplyCard(event.data.payload);
            }
        }
    }
});

Step 2: Handling the Manual Review Workflow
When a suggestion is flagged for review, it must not block the call flow. Instead, it should appear as a non-intrusive notification.

  • UI State: The suggestion appears in a “Pending Review” panel.
  • Agent Action: The agent clicks “Accept” or “Discard”.
  • Audit Trail: Both actions must be logged. If the agent discards a high-confidence suggestion, this is a signal to retrain the model.

The Trap: Overloading the agent with too many “Review” suggestions causes alert fatigue. If the threshold is set too low (e.g., 0.80), the agent will start ignoring the panel entirely.
Architectural Reasoning: We implement a “Silence Period” for low-confidence suggestions. If an agent consistently discards suggestions with confidence scores between 0.80 and 0.90, the system should automatically suppress those suggestions for that agent and log a feedback event for the ML team. This prevents UI clutter and ensures that only high-value interventions are presented.

4. Feedback Loop and Model Retraining

Confidence thresholds are not static. They must be adjusted based on agent behavior.

Step 1: Capturing Implicit Feedback
Every time an agent modifies or discards a Copilot suggestion, capture this event.

  • Event: COPILOT_SUGGESTION_MODIFIED
  • Data: Original suggestion, Agent modification, Confidence score.

Step 2: Analyzing Drift
Use the Genesys Cloud Analytics API to query these events.

GET /api/v2/analytics/conversations/summary?metric=agent_copilot_suggestions&group_by=skill

Step 3: Adjusting Thresholds
If the discard rate for a specific skill exceeds 20%, lower the confidence threshold for auto-accept or increase the threshold for manual review. This is a continuous optimization loop.

The Trap: Ignoring the “False Positive” rate. A high confidence score does not guarantee correctness if the underlying data source is stale.
Architectural Reasoning: We decouple the confidence threshold from the data freshness. If the data source is older than 24 hours, the confidence score is artificially reduced by 0.1 in the Architect logic, forcing manual review. This ensures that even high-confidence AI actions do not rely on outdated information.

Validation, Edge Cases & Troubleshooting

Edge Case 1: The “Hallucination” Spike

  • The Failure Condition: The Copilot starts generating plausible but incorrect data, and the confidence score remains high (e.g., 0.95).
  • The Root Cause: The model is overfitting to recent training data or the system prompt is too permissive.
  • The Solution: Implement a “Grounding” check. Before presenting the suggestion, use a secondary API call to verify the data against the source of truth. If the verification fails, force the confidence score to 0.0. This adds latency but prevents critical errors.

Edge Case 2: Latency in Manual Review

  • The Failure Condition: The agent is on a live call, and the Copilot suggestion for manual review takes 5+ seconds to appear.
  • The Root Cause: The Architect Flow is synchronous and waits for the entire ML inference pipeline to complete before returning control to the UI.
  • The Solution: Use Async Processing. The Architect Flow should trigger the Copilot request via a Task or Message Queue and return control to the agent immediately. The UI component should poll for the result or use WebSockets to receive the suggestion asynchronously. This ensures the agent’s call flow is never blocked by AI latency.

Edge Case 3: Cross-Skill Context Leakage

  • The Failure Condition: A suggestion from the Transaction_Write_Risky skill appears in the Data_Lookup_Read_Only context.
  • The Root Cause: The System Prompt is not sufficiently isolated, or the Conversation Context is being shared across skills.
  • The Solution: Ensure each Skill has a distinct Conversation Context variable. In Architect, explicitly clear the context before invoking a new Skill. Use Scoped Variables to prevent data from one skill leaking into another.

Official References