Building a Custom Bot Performance Dashboard using Analytics API Utterance Data

How-To-Articles
1

Building a Custom Bot Performance Dashboard using Analytics API Utterance Data

What This Guide Covers

This masterclass details the construction of a specialized Bot Performance Dashboard using Genesys Cloud’s Analytics API. By the end of this guide, you will be able to extract raw utterance data, NLU intent matches, and bot exit points to visualize “Confusion Matrices,” “Containment Trends,” and “Top Unrecognized Phrases.” This enables data-driven NLU training and identifies exactly where users are abandoning your bots or being forced into expensive agent escalations.

Prerequisites, Roles & Licensing

Bot analytics require access to interaction metadata and NLU execution logs.

  • Licensing: Genesys Cloud CX 1, 2, or 3.
  • Permissions:
    • Analytics > Conversation Detail > View
    • Architect > Flow > View
  • OAuth Scopes: analytics, conversations.
  • Infrastructure: A middleware layer (Node.js/Python) to poll the API and a visualization tool (Tableau, PowerBI, or a custom React frontend).

The Implementation Deep-Dive

1. Extracting Bot Interaction Metadata

The standard Conversation Detail query returns the “Result” of the bot interaction, but the “Why” is hidden in the botDetails and flowDetails segments of the JSON payload.

The Implementation Pattern:
Query the /api/v2/analytics/conversations/details/query endpoint. Filter for interactions that have a purpose == "customer" and contain a flow segment where flowType == "BOT".

Key Fields to Capture:

  • intentName: The intent the bot matched.
  • confidence: The NLU confidence score (0.0 to 1.0).
  • exitReason: Why the bot session ended (e.g., Escalated, Disconnect, Complete).
  • utteranceText: (Requires specific configuration) The raw text input from the user.

2. Enabling “Raw Utterance” Logging

By default, Genesys Cloud does not store the raw text of every bot utterance in the permanent analytics record for privacy reasons.

The Trap:
Building a dashboard only to find that the utteranceText field is null.
The Solution: You must enable NLU Execution Logging in the Bot Flow settings. For long-term storage and dashboarding, use a Data Action at the end of the bot flow to send the LastUtterance and IntentMatch to your external reporting database.

3. Visualizing the “Confusion Matrix”

A “Confusion Matrix” shows where the bot is misidentifying intents (e.g., users asking about “Pricing” are being matched to “Technical Support”).

Architectural Reasoning:
If you see a high frequency of “Transfer to Agent” events coming from a specific node in the bot flow, it indicates a “Dead-End UI.” The user is likely providing valid input that the bot doesn’t understand, or the bot’s prompt is too ambiguous.

Dashboard KPI Examples:

  • Intent Accuracy: (Successful Matches / Total Utterances).
  • Containment Rate: (Total Bot sessions - Escalated sessions) / Total sessions.
  • Average Turns to Resolution: How many inputs does it take for the bot to reach a Complete state?

4. Implementing the “Top Unrecognized” Alerting System

The most valuable data point is what the bot didn’t understand.

Implementation Step:
Filter your dataset for interactions where intentName == "None" or confidence < 0.60. Group these by utteranceText and count the occurrences. This list becomes your NLU Training Backlog. If 500 people asked “Where is my refund?” and it was unrecognized, you know exactly which intent needs new training utterances.

Validation, Edge Cases & Troubleshooting

Edge Case 1: PII in Utterances

  • The failure condition: Your bot dashboard is inadvertently storing Credit Card numbers or SSNs in the utteranceText field.
  • The root cause: Users providing PII in an unconstrained text input field.
  • The solution: Implement Regex Redaction in your middleware before the data is saved to your reporting database. Mask any string that matches a sensitive pattern (e.g., \d{4}-\d{4}-\d{4}-\d{4}).

Edge Case 2: Multi-Language Skew

  • The failure condition: Containment rates look great in English but terrible in Spanish.
  • The root cause: The NLU model for the secondary language has significantly fewer training utterances.
  • The solution: Add a Language Dimension to your dashboard. This allows you to identify if performance issues are platform-wide or specific to a regional NLU model.

Official References