Resolve State Locks and Drift in Genesys Cloud Routing Queues with Terraform
What You Will Build
- This tutorial resolves a
terraform planfailure caused by a stale state lock on thegenesyscloud_routing_queueresource. - It uses the Genesys Cloud Terraform Provider and the
terraform force-unlockcommand to clear the lock. - It covers the Go-based Terraform provider logic and CLI commands to restore state consistency.
Prerequisites
- Terraform Version: 1.5.0 or later.
- Genesys Cloud Provider Version:
v1.60.0or later (checkregistry.terraform.io/mypurecloud/genesyscloud). - Access Level: You need Genesys Cloud admin rights to view queues and Terraform state file write access.
- Dependencies:
terraforminstalled on your machine.genesyscloudprovider configured in yourmain.tf.
Authentication Setup
The Genesys Cloud Terraform provider handles OAuth authentication automatically via the genesyscloud provider block. You must configure your credentials in the provider block or via environment variables.
# main.tf
terraform {
required_providers {
genesyscloud = {
source = "mypurecloud/genesyscloud"
version = ">= 1.60.0"
}
}
}
provider "genesyscloud" {
# Option 1: Environment Variables (Recommended for CI/CD)
# GENEYS_CLOUD_OAUTH_CLIENT_ID
# GENEYS_CLOUD_OAUTH_CLIENT_SECRET
# GENEYS_CLOUD_OAUTH_GRANT_TYPE
# Option 2: Inline (Not recommended for production)
# oauth_client_id = "your_client_id"
# oauth_client_secret = "your_client_secret"
# oauth_grant_type = "client_credentials"
}
resource "genesyscloud_routing_queue" "support_queue" {
name = "Support Queue"
description = "Main support queue"
enabled = true
# ... other attributes ...
}
Ensure your environment variables are set before running any Terraform commands.
export GENEYS_CLOUD_OAUTH_CLIENT_ID="your_client_id"
export GENEYS_CLOUD_OAUTH_CLIENT_SECRET="your_client_secret"
export GENEYS_CLOUD_OAUTH_GRANT_TYPE="client_credentials"
Implementation
Step 1: Identify the State Lock Error
When you run terraform plan or terraform apply, you may encounter an error indicating that the state file is locked. This often happens if a previous terraform apply was interrupted, killed, or timed out.
Error Output:
Acquiring the state lock. This may take a few moments...
Error: Error acquiring the state lock
Lock Info:
ID: abc123-def456-ghi789
Path: terraform.tfstate
Operation: OperationTypeApply
Who: user@hostname
Version: 1.5.0
Created: 2023-10-27 14:30:00.000000 +0000 UTC
Info: ...
Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again. For most commands, you can disable locking with the "-lock=false"
flag, but this is not recommended.
Do not use -lock=false immediately. This can cause state corruption. Instead, verify if the previous operation actually completed.
Step 2: Verify Resource Status in Genesys Cloud
Before unlocking, check if the genesyscloud_routing_queue resource exists and matches your desired state in Genesys Cloud. Use the Genesys Cloud API to query the queue.
API Endpoint:
GET /api/v2/routing/queues
Python Script to Check Queue Status:
import requests
import os
import json
def get_oauth_token():
"""Retrieve OAuth token from Genesys Cloud."""
client_id = os.getenv("GENEYS_CLOUD_OAUTH_CLIENT_ID")
client_secret = os.getenv("GENEYS_CLOUD_OAUTH_CLIENT_SECRET")
grant_type = os.getenv("GENEYS_CLOUD_OAUTH_GRANT_TYPE", "client_credentials")
url = "https://api.mypurecloud.com/api/v2/oauth/token"
data = {
"client_id": client_id,
"client_secret": client_secret,
"grant_type": grant_type
}
response = requests.post(url, data=data)
if response.status_code != 200:
raise Exception(f"Failed to get OAuth token: {response.text}")
return response.json().get("access_token")
def check_queue_status(queue_name):
"""Check if a queue exists and its current state."""
token = get_oauth_token()
headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json"
}
url = "https://api.mypurecloud.com/api/v2/routing/queues"
# Search for the queue by name
params = {
"search": queue_name,
"pageSize": 25
}
response = requests.get(url, headers=headers, params=params)
if response.status_code == 401:
raise Exception("Unauthorized: Check your OAuth credentials.")
elif response.status_code == 429:
raise Exception("Rate limited: Wait before retrying.")
elif response.status_code != 200:
raise Exception(f"API Error: {response.status_code} - {response.text}")
data = response.json()
queues = data.get("entities", [])
for queue in queues:
if queue.get("name") == queue_name:
return {
"id": queue.get("id"),
"name": queue.get("name"),
"enabled": queue.get("enabled"),
"wrapUpTimerRequired": queue.get("wrapUpTimerRequired")
}
return None
if __name__ == "__main__":
queue_name = "Support Queue"
status = check_queue_status(queue_name)
if status:
print(json.dumps(status, indent=2))
else:
print(f"Queue '{queue_name}' not found.")
Run this script to confirm the queue exists and its attributes match your main.tf configuration. If the queue is missing or misconfigured, the previous terraform apply likely failed partially.
Step 3: Force Unlock the State
If you confirm that the previous operation did not leave the state in an inconsistent condition (or if you are certain the operation failed completely), you can force unlock the state.
Command:
terraform force-unlock <LOCK_ID>
Replace <LOCK_ID> with the ID from the error message (e.g., abc123-def456-ghi789).
Example:
terraform force-unlock abc123-def456-ghi789
Expected Output:
Releasing the state lock. This may take a few moments...
Success! The state has been unlocked.
Step 4: Refresh State and Plan
After unlocking, refresh the state to ensure Terraform has the latest remote state.
Command:
terraform refresh
Then, run terraform plan to see if there is any drift.
Command:
terraform plan
If the plan shows no changes, the issue is resolved. If it shows changes, apply them.
Complete Working Example
Here is a complete workflow to handle the state lock issue.
1. Check Queue Status (Python):
# check_queue.py
import requests
import os
import json
import sys
def get_oauth_token():
client_id = os.getenv("GENEYS_CLOUD_OAUTH_CLIENT_ID")
client_secret = os.getenv("GENEYS_CLOUD_OAUTH_CLIENT_SECRET")
grant_type = os.getenv("GENEYS_CLOUD_OAUTH_GRANT_TYPE", "client_credentials")
url = "https://api.mypurecloud.com/api/v2/oauth/token"
data = {
"client_id": client_id,
"client_secret": client_secret,
"grant_type": grant_type
}
response = requests.post(url, data=data)
if response.status_code != 200:
raise Exception(f"Failed to get OAuth token: {response.text}")
return response.json().get("access_token")
def check_queue_status(queue_name):
token = get_oauth_token()
headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json"
}
url = "https://api.mypurecloud.com/api/v2/routing/queues"
params = {
"search": queue_name,
"pageSize": 25
}
response = requests.get(url, headers=headers, params=params)
if response.status_code == 401:
raise Exception("Unauthorized: Check your OAuth credentials.")
elif response.status_code == 429:
raise Exception("Rate limited: Wait before retrying.")
elif response.status_code != 200:
raise Exception(f"API Error: {response.status_code} - {response.text}")
data = response.json()
queues = data.get("entities", [])
for queue in queues:
if queue.get("name") == queue_name:
return {
"id": queue.get("id"),
"name": queue.get("name"),
"enabled": queue.get("enabled"),
"wrapUpTimerRequired": queue.get("wrapUpTimerRequired")
}
return None
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python check_queue.py <queue_name>")
sys.exit(1)
queue_name = sys.argv[1]
status = check_queue_status(queue_name)
if status:
print(json.dumps(status, indent=2))
else:
print(f"Queue '{queue_name}' not found.")
2. Bash Script to Handle Lock:
#!/bin/bash
# resolve_lock.sh
LOCK_ID="abc123-def456-ghi789" # Replace with your lock ID
QUEUE_NAME="Support Queue"
echo "Checking queue status..."
python3 check_queue.py "$QUEUE_NAME"
if [ $? -eq 0 ]; then
echo "Queue status retrieved successfully."
else
echo "Failed to retrieve queue status. Aborting."
exit 1
fi
echo "Forcing unlock with ID: $LOCK_ID"
terraform force-unlock "$LOCK_ID"
if [ $? -eq 0 ]; then
echo "Unlock successful. Refreshing state..."
terraform refresh
echo "Running plan..."
terraform plan
else
echo "Failed to unlock state."
exit 1
fi
Common Errors & Debugging
Error: Lock ID Not Found
What causes it:
The lock ID provided does not exist in the state file. This can happen if the lock was already released by another process or if the state file was manually edited.
How to fix it:
Run terraform state pull to inspect the state file manually. Look for the lock section. If no lock exists, you can proceed with terraform plan without unlocking.
Code showing the fix:
terraform state pull | grep -A 10 '"lock"'
If no lock is found, run:
terraform plan
Error: 409 Conflict During Apply
What causes it:
After unlocking, if you run terraform apply, you may get a 409 Conflict error if the resource in Genesys Cloud has been modified outside of Terraform.
How to fix it:
Run terraform refresh to update the local state with the current remote state. Then, run terraform plan to see the differences. If the differences are expected, apply them. If not, investigate the manual changes in Genesys Cloud.
Code showing the fix:
terraform refresh
terraform plan
terraform apply
Error: State File Corrupted
What causes it:
If you used -lock=false and the operation failed, the state file may be corrupted.
How to fix it:
Restore the state file from a backup. If no backup exists, you may need to import the existing resources into Terraform.
Code showing the fix:
# Import existing queue
terraform import genesyscloud_routing_queue.support_queue <QUEUE_ID>