Building a Custom NLU Utterance Training and Validation Pipeline in Architect

How-To-Articles
1

Building a Custom NLU Utterance Training and Validation Pipeline in Architect

What This Guide Covers

  • Designing a continuous integration/continuous deployment (CI/CD) methodology for Natural Language Understanding (NLU) domains within Genesys Cloud Architect.
  • Extracting raw customer utterances via the Analytics API to systematically identify false positives and false negatives in your bot’s intent classification.
  • The end result is a highly disciplined training pipeline where bot intent accuracy is quantified and systematically improved using ground-truth data, rather than relying on developers subjectively guessing what a customer might say.

Prerequisites, Roles & Licensing

  • Licensing: Genesys Cloud CX 2 or 3 (Digital).
  • Permissions: Architect > Flow > Edit, Analytics > Conversation Detail > View.
  • Infrastructure: A backend script (Python) or a data warehouse (e.g., Snowflake/AWS Athena) to aggregate the Analytics API data.

The Implementation Deep-Dive

1. The Trap of “Synthetic” Utterance Training

When developers build a new Bot Flow (e.g., an “Address Change” intent), they typically train it by typing what they think a customer would say:

  • “I want to change my address”
  • “Update my location”
  • “I moved”

The Trap:
Customers do not speak like developers. A customer actually says: “Hi, yeah, my husband and I just bought a house in Florida and I need to make sure the bill goes there now.” Because the developer didn’t train the model on this verbose, conversational structure, the bot fails and triggers the CatchAll path.

2. Extracting Ground-Truth Utterances

To fix this, you must train your model using historical, real-world utterances. Genesys Cloud stores every utterance evaluated by a bot, along with the intent it matched (or failed to match) and the confidence score.

Implementation Steps:

  1. The Analytics Query: Use the POST /api/v2/analytics/conversations/details/query endpoint.
  2. Filter for conversations that hit your specific Bot Flow ID.
  3. Parsing the Payload: In the JSON response, navigate to participants > sessions > metrics. Look for the nluIntentEvaluated events.
  4. Extract the utterance text, the intentName it matched, and the confidence score.
  5. Aggregate this data over the last 30 days into a CSV file.

3. The Validation Pipeline

Once you have the CSV, you must establish a validation workflow. Never blindly feed raw customer utterances back into the NLU training model.

Architectural Reasoning:
If a customer says “I want to cancel my account because your service is terrible,” and you add that exact string to the Cancel_Account intent training data, you are overfitting the model. The model will start associating the phrase “terrible service” heavily with the cancel intent, potentially causing false positives when a customer calls to complain about terrible service but wants to stay.

Implementation Steps (The Review Process):

  1. Identify “No Match” Utterances: Filter your CSV for utterances where intentName was blank or matched the CatchAll.
  2. Review and Categorize: Have a human analyst review the top 50 most frequent “No Match” phrases.
  3. Sanitize: Strip out personal data, names, and highly specific context. Change “I need my bill sent to Florida” to “I need my bill sent to [State]”.
  4. Identify “False Positives”: Filter your CSV for utterances where the confidence score was between 0.40 and 0.65 (a weak match). Did the bot guess correctly? If not, the NLU model boundaries are blurred.

4. Injecting Training Data and Retraining

After sanitizing the new utterances, you must upload them to Architect.

Implementation Steps:

  1. Navigate to Architect > Natural Language Understanding. Open your Domain.
  2. Go to the intent (e.g., Update_Address).
  3. Manually add the new sanitized utterances.
  4. Crucial Step: Run the Optimizer in Architect. The NLU Optimizer evaluates the new training data against all other intents in the domain.
  5. If the Optimizer flags a conflict (e.g., the new utterance overlaps too heavily with Billing_Question), you must resolve the overlap before publishing. Do this by making the utterances more distinct or creating a disambiguation bot menu.

Validation, Edge Cases & Troubleshooting

Edge Case 1: The “Greeting” Contamination

  • The Failure Condition: Customers frequently say: “Hi there, how are you? I need to pay my bill.” Because you added this entire phrase to the Pay_Bill intent, the model starts thinking “Hi there” is a strong indicator of paying a bill. Later, a customer says “Hi there, I need to cancel”, and the bot routes them to billing.
  • The Root Cause: NLU models learn patterns. If common greetings are included in intent training phrases, the model over-indexes on the greeting.
  • The Solution: Strip all greetings and pleasantries from your training utterances before uploading them to Architect. The NLU model should only be trained on the core action (e.g., “pay my bill”, “cancel account”).

Edge Case 2: Multi-Intent Utterances

  • The Failure Condition: A customer says: “I need to pay my bill and then change my address.” The bot gets confused, assigns a low confidence to both, and fails.
  • The Root Cause: Genesys Cloud native bots assign a single intent per utterance. They do not natively handle multi-intent complex payloads perfectly in a single pass without advanced slot-filling configuration.
  • The Solution: In your NLU training data, do not train intents on multi-intent utterances. Let the bot fail to a disambiguation prompt: “It sounds like you need help with Billing and Account Changes. Which one would you like to do first?” Use the NLU evaluation events to identify how often customers ask two questions at once, and if it exceeds 10%, re-architect your bot greeting to explicitly ask for one issue at a time.

Official References