Resolve State Locks and Drift in Genesys Cloud Routing Queues with Terraform

Resolve State Locks and Drift in Genesys Cloud Routing Queues with Terraform

What You Will Build

  • This tutorial resolves a terraform plan failure caused by a stale state lock on the genesyscloud_routing_queue resource.
  • It uses the Genesys Cloud Terraform Provider and the terraform force-unlock command to clear the lock.
  • It covers the Go-based Terraform provider logic and CLI commands to restore state consistency.

Prerequisites

  • Terraform Version: 1.5.0 or later.
  • Genesys Cloud Provider Version: v1.60.0 or later (check registry.terraform.io/mypurecloud/genesyscloud).
  • Access Level: You need Genesys Cloud admin rights to view queues and Terraform state file write access.
  • Dependencies:
    • terraform installed on your machine.
    • genesyscloud provider configured in your main.tf.

Authentication Setup

The Genesys Cloud Terraform provider handles OAuth authentication automatically via the genesyscloud provider block. You must configure your credentials in the provider block or via environment variables.

# main.tf
terraform {
  required_providers {
    genesyscloud = {
      source  = "mypurecloud/genesyscloud"
      version = ">= 1.60.0"
    }
  }
}

provider "genesyscloud" {
  # Option 1: Environment Variables (Recommended for CI/CD)
  # GENEYS_CLOUD_OAUTH_CLIENT_ID
  # GENEYS_CLOUD_OAUTH_CLIENT_SECRET
  # GENEYS_CLOUD_OAUTH_GRANT_TYPE

  # Option 2: Inline (Not recommended for production)
  # oauth_client_id     = "your_client_id"
  # oauth_client_secret = "your_client_secret"
  # oauth_grant_type    = "client_credentials"
}

resource "genesyscloud_routing_queue" "support_queue" {
  name        = "Support Queue"
  description = "Main support queue"
  enabled     = true
  # ... other attributes ...
}

Ensure your environment variables are set before running any Terraform commands.

export GENEYS_CLOUD_OAUTH_CLIENT_ID="your_client_id"
export GENEYS_CLOUD_OAUTH_CLIENT_SECRET="your_client_secret"
export GENEYS_CLOUD_OAUTH_GRANT_TYPE="client_credentials"

Implementation

Step 1: Identify the State Lock Error

When you run terraform plan or terraform apply, you may encounter an error indicating that the state file is locked. This often happens if a previous terraform apply was interrupted, killed, or timed out.

Error Output:

Acquiring the state lock. This may take a few moments...
Error: Error acquiring the state lock

Lock Info:
  ID:        abc123-def456-ghi789
  Path:      terraform.tfstate
  Operation: OperationTypeApply
  Who:       user@hostname
  Version:   1.5.0
  Created:   2023-10-27 14:30:00.000000 +0000 UTC
  Info:      ...

Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again. For most commands, you can disable locking with the "-lock=false"
flag, but this is not recommended.

Do not use -lock=false immediately. This can cause state corruption. Instead, verify if the previous operation actually completed.

Step 2: Verify Resource Status in Genesys Cloud

Before unlocking, check if the genesyscloud_routing_queue resource exists and matches your desired state in Genesys Cloud. Use the Genesys Cloud API to query the queue.

API Endpoint:
GET /api/v2/routing/queues

Python Script to Check Queue Status:

import requests
import os
import json

def get_oauth_token():
    """Retrieve OAuth token from Genesys Cloud."""
    client_id = os.getenv("GENEYS_CLOUD_OAUTH_CLIENT_ID")
    client_secret = os.getenv("GENEYS_CLOUD_OAUTH_CLIENT_SECRET")
    grant_type = os.getenv("GENEYS_CLOUD_OAUTH_GRANT_TYPE", "client_credentials")

    url = "https://api.mypurecloud.com/api/v2/oauth/token"
    data = {
        "client_id": client_id,
        "client_secret": client_secret,
        "grant_type": grant_type
    }

    response = requests.post(url, data=data)
    if response.status_code != 200:
        raise Exception(f"Failed to get OAuth token: {response.text}")
    
    return response.json().get("access_token")

def check_queue_status(queue_name):
    """Check if a queue exists and its current state."""
    token = get_oauth_token()
    headers = {
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json"
    }

    url = "https://api.mypurecloud.com/api/v2/routing/queues"
    
    # Search for the queue by name
    params = {
        "search": queue_name,
        "pageSize": 25
    }

    response = requests.get(url, headers=headers, params=params)
    
    if response.status_code == 401:
        raise Exception("Unauthorized: Check your OAuth credentials.")
    elif response.status_code == 429:
        raise Exception("Rate limited: Wait before retrying.")
    elif response.status_code != 200:
        raise Exception(f"API Error: {response.status_code} - {response.text}")

    data = response.json()
    queues = data.get("entities", [])
    
    for queue in queues:
        if queue.get("name") == queue_name:
            return {
                "id": queue.get("id"),
                "name": queue.get("name"),
                "enabled": queue.get("enabled"),
                "wrapUpTimerRequired": queue.get("wrapUpTimerRequired")
            }
    
    return None

if __name__ == "__main__":
    queue_name = "Support Queue"
    status = check_queue_status(queue_name)
    if status:
        print(json.dumps(status, indent=2))
    else:
        print(f"Queue '{queue_name}' not found.")

Run this script to confirm the queue exists and its attributes match your main.tf configuration. If the queue is missing or misconfigured, the previous terraform apply likely failed partially.

Step 3: Force Unlock the State

If you confirm that the previous operation did not leave the state in an inconsistent condition (or if you are certain the operation failed completely), you can force unlock the state.

Command:

terraform force-unlock <LOCK_ID>

Replace <LOCK_ID> with the ID from the error message (e.g., abc123-def456-ghi789).

Example:

terraform force-unlock abc123-def456-ghi789

Expected Output:

Releasing the state lock. This may take a few moments...
Success! The state has been unlocked.

Step 4: Refresh State and Plan

After unlocking, refresh the state to ensure Terraform has the latest remote state.

Command:

terraform refresh

Then, run terraform plan to see if there is any drift.

Command:

terraform plan

If the plan shows no changes, the issue is resolved. If it shows changes, apply them.

Complete Working Example

Here is a complete workflow to handle the state lock issue.

1. Check Queue Status (Python):

# check_queue.py
import requests
import os
import json
import sys

def get_oauth_token():
    client_id = os.getenv("GENEYS_CLOUD_OAUTH_CLIENT_ID")
    client_secret = os.getenv("GENEYS_CLOUD_OAUTH_CLIENT_SECRET")
    grant_type = os.getenv("GENEYS_CLOUD_OAUTH_GRANT_TYPE", "client_credentials")

    url = "https://api.mypurecloud.com/api/v2/oauth/token"
    data = {
        "client_id": client_id,
        "client_secret": client_secret,
        "grant_type": grant_type
    }

    response = requests.post(url, data=data)
    if response.status_code != 200:
        raise Exception(f"Failed to get OAuth token: {response.text}")
    
    return response.json().get("access_token")

def check_queue_status(queue_name):
    token = get_oauth_token()
    headers = {
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json"
    }

    url = "https://api.mypurecloud.com/api/v2/routing/queues"
    params = {
        "search": queue_name,
        "pageSize": 25
    }

    response = requests.get(url, headers=headers, params=params)
    
    if response.status_code == 401:
        raise Exception("Unauthorized: Check your OAuth credentials.")
    elif response.status_code == 429:
        raise Exception("Rate limited: Wait before retrying.")
    elif response.status_code != 200:
        raise Exception(f"API Error: {response.status_code} - {response.text}")

    data = response.json()
    queues = data.get("entities", [])
    
    for queue in queues:
        if queue.get("name") == queue_name:
            return {
                "id": queue.get("id"),
                "name": queue.get("name"),
                "enabled": queue.get("enabled"),
                "wrapUpTimerRequired": queue.get("wrapUpTimerRequired")
            }
    
    return None

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python check_queue.py <queue_name>")
        sys.exit(1)
        
    queue_name = sys.argv[1]
    status = check_queue_status(queue_name)
    if status:
        print(json.dumps(status, indent=2))
    else:
        print(f"Queue '{queue_name}' not found.")

2. Bash Script to Handle Lock:

#!/bin/bash
# resolve_lock.sh

LOCK_ID="abc123-def456-ghi789" # Replace with your lock ID
QUEUE_NAME="Support Queue"

echo "Checking queue status..."
python3 check_queue.py "$QUEUE_NAME"

if [ $? -eq 0 ]; then
    echo "Queue status retrieved successfully."
else
    echo "Failed to retrieve queue status. Aborting."
    exit 1
fi

echo "Forcing unlock with ID: $LOCK_ID"
terraform force-unlock "$LOCK_ID"

if [ $? -eq 0 ]; then
    echo "Unlock successful. Refreshing state..."
    terraform refresh
    echo "Running plan..."
    terraform plan
else
    echo "Failed to unlock state."
    exit 1
fi

Common Errors & Debugging

Error: Lock ID Not Found

What causes it:
The lock ID provided does not exist in the state file. This can happen if the lock was already released by another process or if the state file was manually edited.

How to fix it:
Run terraform state pull to inspect the state file manually. Look for the lock section. If no lock exists, you can proceed with terraform plan without unlocking.

Code showing the fix:

terraform state pull | grep -A 10 '"lock"'

If no lock is found, run:

terraform plan

Error: 409 Conflict During Apply

What causes it:
After unlocking, if you run terraform apply, you may get a 409 Conflict error if the resource in Genesys Cloud has been modified outside of Terraform.

How to fix it:
Run terraform refresh to update the local state with the current remote state. Then, run terraform plan to see the differences. If the differences are expected, apply them. If not, investigate the manual changes in Genesys Cloud.

Code showing the fix:

terraform refresh
terraform plan
terraform apply

Error: State File Corrupted

What causes it:
If you used -lock=false and the operation failed, the state file may be corrupted.

How to fix it:
Restore the state file from a backup. If no backup exists, you may need to import the existing resources into Terraform.

Code showing the fix:

# Import existing queue
terraform import genesyscloud_routing_queue.support_queue <QUEUE_ID>

Official References