Architecting Weighted Scoring Models for Multi-Dimensional Agent Performance Evaluation

How-To-Articles
1

Architecting Weighted Scoring Models for Multi-Dimensional Agent Performance Evaluation

What This Guide Covers

This guide details the architectural implementation of dynamic, weighted scoring models within Genesys Cloud CX and NICE CXone to evaluate agent performance across multiple dimensions (QA, WFM, CSAT). You will configure the data aggregation pipelines, define the weighting algorithms, and build the visualization layer that transforms raw metric streams into a single, actionable performance index.

Prerequisites, Roles & Licensing

Licensing Requirements

  • Genesys Cloud CX: CX 2 or CX 3 license is mandatory for access to Quality Management (QM) and Workforce Engagement Management (WEM). For real-time dashboarding and custom API integrations, CX 3 is recommended.
  • NICE CXone: CXone Connect with Quality Management and Workforce Management add-ons. The “Insights” module is required for advanced reporting and custom scorecard creation.

Required Permissions

  • Genesys Cloud:
    • Quality > Quality Management > Edit
    • Reporting > Reporting > View
    • User Management > User Management > Edit
    • Integrations > Integration > Edit (if using outbound APIs)
  • NICE CXone:
    • Quality > Quality Management > Create/Edit
    • Reporting > Reporting > Create/Edit
    • Admin > User Management > Edit

External Dependencies

  • Data Warehouse/SIEM: Optional but recommended for long-term trend analysis (e.g., Snowflake, BigQuery).
  • HRIS Integration: For correlating performance scores with tenure or training completion status.

The Implementation Deep-Dive

1. Define the Dimensional Taxonomy and Data Sources

Before configuring the scoring engine, you must map the specific metrics that constitute “performance.” A common architectural failure is treating all metrics as equally weighted or equally sourced. You must separate synchronous interaction metrics from asynchronous behavioral metrics.

The Taxonomy Structure

We divide performance into three primary dimensions:

  1. Quality Assurance (QA): Compliance, empathy, accuracy, and adherence to script.
  2. Workforce Management (WFM): Adherence, shrinkage, and availability.
  3. Customer Experience (CX): CSAT, NPS, and CES.

Data Source Mapping

  • QA Data: Sourced from the Quality Management module. This data is discrete, event-driven, and often sampled (e.g., 5% of calls).
  • WFM Data: Sourced from the WEM module. This data is continuous, time-series based, and high-volume.
  • CX Data: Sourced from Survey Management or external CRM webhooks. This data is low-volume and highly variable in latency.

The Trap: Aggregating sampled QA data with continuous WFM data without normalization. If you score an agent on 10 QA evaluations and 10,000 minutes of adherence, the WFM metric will drown out the QA metric unless you apply inverse frequency weighting.

Architectural Reasoning

We do not store the final “Performance Score” in the primary database. Instead, we calculate it in the reporting layer or via a middleware function. This ensures that if the weighting algorithm changes (e.g., increasing QA weight from 30% to 40%), historical data remains intact and re-calculable.

2. Configure the Genesys Cloud CX Quality Scorecard

In Genesys Cloud, the Quality Management module allows for granular scorecard configuration. You must structure the scorecard to output numeric values that can be mathematically aggregated.

Step 2.1: Create the Scorecard

Navigate to Admin > Quality > Scorecards. Create a new scorecard named Global_Performance_Scorecard.

Critical Configuration:

  • Set the Scoring Type to Points.
  • Ensure every section has a defined maximum point value. Avoid Pass/Fail only sections unless you map them to points (e.g., Pass = 10, Fail = 0).

Step 2.2: Define Weighted Sections

Do not rely on the default equal weighting. Use the Section Weight field.

Example Configuration:

  • Compliance: Weight 40, Max Points 100.
  • Empathy: Weight 30, Max Points 100.
  • Resolution: Weight 30, Max Points 100.

The Trap: Using relative weights that do not sum to 100 or 1.0. Genesys Cloud normalizes these internally, but if you export this data via API for external calculation, you must ensure the weights are absolute. If you export raw points, you must manually apply the weights in your aggregation layer.

Step 2.3: API Payload for Scorecard Creation

To automate this, use the Genesys Cloud API.

POST /api/v2/quality/scorecards
Authorization: Bearer <access_token>
Content-Type: application/json

{
  "name": "Global_Performance_Scorecard",
  "description": "Master scorecard for multi-dimensional evaluation",
  "sections": [
    {
      "name": "Compliance",
      "description": "Regulatory adherence",
      "weight": 0.4,
      "items": [
        {
          "name": "ID Verification",
          "description": "Agent verified customer ID",
          "type": "boolean",
          "passFail": false,
          "weight": 1.0
        }
      ]
    },
    {
      "name": "Empathy",
      "description": "Customer emotional connection",
      "weight": 0.3,
      "items": [
        {
          "name": "Active Listening",
          "description": "Agent demonstrated active listening",
          "type": "scale",
          "minValue": 1,
          "maxValue": 5,
          "weight": 1.0
        }
      ]
    }
  ]
}

3. Aggregate WFM Metrics via Reporting API

WFM metrics are not stored in the Quality module. You must pull them from the Reporting API. The key metrics are Adherence and Shrinkage.

Step 3.1: Identify the Correct Report Type

Use the workforce-management report type. Specifically, you need the agent-adherence and agent-shrinkage report types.

The Trap: Using agent-productivity for adherence. Productivity includes after-call work (ACW) and other states that may not reflect strict schedule adherence. For performance evaluation, strict adherence to the schedule is the standard.

Step 3.2: Query Construction

Construct a query that returns daily adherence percentages for each agent.

POST /api/v2/analytics/reporting/query
Authorization: Bearer <access_token>
Content-Type: application/json

{
  "reportType": "workforce-management",
  "reportId": "workforce-management-agent-adherence",
  "dateFrom": "2023-10-01T00:00:00.000Z",
  "dateTo": "2023-10-31T23:59:59.999Z",
  "groupBy": [
    "agentId"
  ],
  "metrics": [
    "adherence"
  ],
  "filters": [
    {
      "type": "string",
      "path": "agentId",
      "operator": "in",
      "values": ["<agent_id_1>", "<agent_id_2>"]
    }
  ]
}

Architectural Reasoning

We pull daily adherence. Why daily? Because monthly averages hide volatility. An agent with 95% adherence on 28 days and 50% on 2 days has the same average as an agent with 90% consistency. For performance scoring, consistency matters. We will calculate the standard deviation of daily adherence in the next step.

4. Implement the Multi-Dimensional Weighting Algorithm

This is the core of the architecture. You must decide on the weighting strategy. A common industry standard is the Balanced Scorecard Approach:

  • QA Score: 40%
  • WFM Adherence: 30%
  • CSAT/NPS: 30%

Step 4.1: Normalization

All metrics must be normalized to a 0-100 scale before weighting.

  • QA: Already 0-100.
  • WFM Adherence: Already 0-100.
  • CSAT: Convert to a percentage. If using a 1-5 scale, (Average Score - 1) / 4 * 100.

Step 4.2: The Aggregation Function

Use a serverless function (AWS Lambda, Azure Function) or a scheduled Python script to aggregate the data.

Python Pseudocode for Aggregation:

def calculate_performance_score(qa_score, adherence_score, csat_score):
    """
    Calculates the weighted performance score.
    Inputs are normalized 0-100.
    """
    qa_weight = 0.40
    wfm_weight = 0.30
    cx_weight = 0.30

    # Apply weights
    weighted_qa = qa_score * qa_weight
    weighted_wfm = adherence_score * wfm_weight
    weighted_cx = csat_score * cx_weight

    # Sum
    total_score = weighted_qa + weighted_wfm + weighted_cx

    # Round to 2 decimal places
    return round(total_score, 2)

def adjust_for_volatility(adherence_daily_scores):
    """
    Penalizes agents with high variance in adherence.
    """
    if len(adherence_daily_scores) < 5:
        return 0  # Insufficient data

    import statistics
    mean_adherence = statistics.mean(adherence_daily_scores)
    stdev_adherence = statistics.stdev(adherence_daily_scores)

    # Penalty: Subtract 1 point for every 1% standard deviation above 5%
    penalty = max(0, (stdev_adherence - 5) * 1)

    return mean_adherence - penalty

The Trap: Ignoring data sparsity. If an agent has no QA scores for the month, their QA component should not be 0. It should be null or excluded from the denominator. If you set it to 0, you penalize the agent for not being evaluated, rather than poor performance.

Solution: Implement a “Minimum Evaluation Threshold.” If an agent has fewer than 3 QA evaluations, their QA score is null. The total score is then calculated using only the available dimensions, with weights re-normalized.

def normalize_weights(qa_valid, wfm_valid, cx_valid):
    total_valid = sum([qa_valid, wfm_valid, cx_valid])
    if total_valid == 0:
        return None

    # Re-normalize weights based on available data
    qa_weight = (0.40 / total_valid) * 100 if qa_valid else 0
    wfm_weight = (0.30 / total_valid) * 100 if wfm_valid else 0
    cx_weight = (0.30 / total_valid) * 100 if cx_valid else 0

    return qa_weight, wfm_weight, cx_weight

5. Visualize and Act on the Data

The final step is to present this data to agents and managers. In Genesys Cloud, you can use Dashboards or Custom Widgets.

Step 5.1: Create a Custom Dashboard Widget

Use the Genesys Cloud Dashboard API to create a widget that displays the aggregated score.

The Trap: Displaying only the final score. Agents need to know where they are losing points. You must display the breakdown (QA, WFM, CX) alongside the total.

Step 5.2: Configure Alerts for Low Performance

Set up automated alerts via the Notifications module or API.

POST /api/v2/notifications/topics
Authorization: Bearer <access_token>
Content-Type: application/json

{
  "name": "Low_Performance_Alert",
  "description": "Alert when performance score drops below 70",
  "filters": [
    {
      "path": "performance_score",
      "operator": "less_than",
      "value": 70
    }
  ],
  "actions": [
    {
      "type": "email",
      "to": ["manager@example.com"]
    }
  ]
}

Architectural Reasoning

We use a threshold of 70. This is arbitrary and should be calibrated based on your organization’s historical data. The key is consistency. The alert must be triggered by the aggregated score, not individual metrics, to provide a holistic view.

Validation, Edge Cases & Troubleshooting

Edge Case 1: The “New Hire” Data Void

The Failure Condition: A new hire starts. They have zero QA scores, zero CSAT, and partial WFM data. The system returns NaN or 0 for their performance score.
The Root Cause: The aggregation function divides by zero or applies weights to null values.
The Solution: Implement a “Probationary Mode.” For the first 30 days, only WFM adherence is calculated. The final score is displayed as “WFM Only” with a disclaimer. Once QA evaluations begin, the model transitions to the full weighted score.

Edge Case 2: The “Outlier” CSAT Spike

The Failure Condition: An agent receives one 1-star CSAT out of 100 5-star ratings. Their average drops significantly, skewing the CX dimension.
The Root Cause: Using arithmetic mean for skewed distributions.
The Solution: Use a trimmed mean for CSAT. Exclude the top and bottom 5% of responses before calculating the average. Alternatively, use the median.

def trimmed_mean(data, trim_percentage=0.05):
    n = len(data)
    trim_count = int(n * trim_percentage)
    sorted_data = sorted(data)
    trimmed_data = sorted_data[trim_count:-trim_count]
    return sum(trimmed_data) / len(trimmed_data)

Edge Case 3: Cross-Channel Disparity

The Failure Condition: An agent handles both voice and chat. Voice QA scores are 85, Chat QA scores are 95. The system averages them to 90. However, the agent spends 80% of their time on chat.
The Root Cause: Equal weighting of QA scores regardless of channel volume.
The Solution: Weight QA scores by channel volume. If an agent has 100 chat evaluations and 10 voice evaluations, the chat score should carry 90% of the QA weight.

def volume_weighted_qa(chat_score, chat_count, voice_score, voice_count):
    total_count = chat_count + voice_count
    if total_count == 0:
        return None

    chat_weight = chat_count / total_count
    voice_weight = voice_count / total_count

    return (chat_score * chat_weight) + (voice_score * voice_weight)

Official References