Implementing NICE Cognigy Webhook Retry Logic via REST API with Python

Implementing NICE Cognigy Webhook Retry Logic via REST API with Python

What You Will Build

You will build a Python module that configures and manages webhook retry policies for NICE Cognigy bots using atomic PATCH operations, implements exponential backoff with jitter, validates payloads against engine constraints, tracks delivery metrics, and routes failed events to a dead letter queue. You will use the Cognigy REST API with the httpx library for precise control over retry scheduling and idempotency verification. You will write the implementation in Python 3.10+.

Prerequisites

  • OAuth2 Client Credentials flow or Cognigy API Key
  • Required scopes: bot:webhooks:manage, bot:webhooks:read
  • Cognigy API v2
  • Python 3.10+
  • External dependencies: httpx, pydantic, pytz, uuid

Authentication Setup

Cognigy uses standard OAuth2 client credentials flow for server-to-server API access. You must fetch a bearer token and cache it with a safety margin to prevent mid-request expiration. The following class handles token acquisition, caching, and automatic refresh logic.

import httpx
import time
from datetime import datetime, timezone, timedelta
from typing import Optional

class CognigyAuth:
    def __init__(self, client_id: str, client_secret: str, auth_url: str, scopes: str):
        self.client_id = client_id
        self.client_secret = client_secret
        self.auth_url = auth_url
        self.scopes = scopes
        self.token: Optional[str] = None
        self.expires_at: Optional[float] = None

    def get_token(self) -> str:
        # Return cached token if valid
        if self.token and self.expires_at and time.time() < self.expires_at:
            return self.token
            
        # Fetch new token
        response = httpx.post(
            self.auth_url,
            data={"grant_type": "client_credentials", "scope": self.scopes},
            auth=(self.client_id, self.client_secret),
            timeout=10.0
        )
        response.raise_for_status()
        token_data = response.json()
        self.token = token_data["access_token"]
        # Subtract 60 seconds as a safety buffer
        self.expires_at = time.time() + token_data["expires_in"] - 60
        return self.token

Implementation

Step 1: Schema Validation and Payload Construction

You must validate retry configurations against Cognigy bot engine constraints before sending them to the API. The engine enforces strict limits on maximum retry windows and attempt counts to prevent indefinite looping failures. You will use Pydantic to enforce these constraints and construct the retry payload with explicit webhook ID references.

from pydantic import BaseModel, field_validator
from typing import Literal

class RetryConfig(BaseModel):
    webhook_id: str
    max_attempts: int = 3
    backoff_strategy: Literal["exponential", "linear"] = "exponential"
    base_delay_ms: int = 1000
    max_delay_ms: int = 30000
    max_retry_window_seconds: int = 3600
    idempotency_key: str

    @field_validator("max_attempts")
    @classmethod
    def validate_max_attempts(cls, v: int) -> int:
        if not (1 <= v <= 10):
            raise ValueError("Cognigy engine constraint: max_attempts must be between 1 and 10.")
        return v

    @field_validator("max_retry_window_seconds")
    @classmethod
    def validate_window(cls, v: int) -> int:
        if v > 86400:
            raise ValueError("Cognigy engine constraint: max_retry_window_seconds cannot exceed 86400.")
        return v

    def to_api_payload(self) -> dict:
        return {
            "retryPolicy": {
                "enabled": True,
                "maxAttempts": self.max_attempts,
                "backoffStrategy": self.backoff_strategy,
                "baseDelayMs": self.base_delay_ms,
                "maxDelayMs": self.max_delay_ms,
                "maxRetryWindowSeconds": self.max_retry_window_seconds
            }
        }

Step 2: Backoff Strategy Matrix and Idempotency Verification

Retry scheduling requires a deterministic backoff matrix with jitter to prevent thundering herd problems during bot scaling. You will also implement payload idempotency verification to prevent data duplication when the same retry event is processed multiple times.

import random
import hashlib
import json
from typing import Dict, Any

def calculate_backoff_ms(attempt: int, base_delay: int, max_delay: int, strategy: str) -> int:
    """Calculate delay with strategy matrix and jitter."""
    if strategy == "exponential":
        delay = base_delay * (2 ** (attempt - 1))
    elif strategy == "linear":
        delay = base_delay * attempt
    else:
        delay = base_delay
        
    # Add uniform jitter (0 to 10% of calculated delay)
    jitter = random.uniform(0, 0.1 * delay)
    return min(int(delay + jitter), max_delay)

def verify_idempotency(payload: Dict[str, Any], stored_hashes: Dict[str, str]) -> bool:
    """Verify payload idempotency using SHA-256 hash comparison."""
    payload_bytes = json.dumps(payload, sort_keys=True).encode("utf-8")
    current_hash = hashlib.sha256(payload_bytes).hexdigest()
    
    # Check against stored hashes to prevent duplicate processing
    for key, existing_hash in stored_hashes.items():
        if current_hash == existing_hash:
            return False  # Duplicate detected
    return True  # Unique payload

Step 3: Atomic PATCH Operations and Retry Scheduling

You will schedule retry policies using atomic PATCH operations against the Cognigy webhook endpoint. The request must include the Idempotency-Key header, proper OAuth scopes, and explicit error handling for 429 rate limits and 409 conflict states. Pagination is handled when listing webhooks to ensure full coverage during automated bot management.

import logging
import time

logger = logging.getLogger("cognigy_webhook_retrier")

class CognigyWebhookRetrier:
    def __init__(self, base_url: str, auth: CognigyAuth):
        self.base_url = base_url
        self.auth = auth
        self.client = httpx.Client(timeout=httpx.Timeout(30.0, connect=10.0))
        self.processed_hashes: Dict[str, str] = {}

    def _handle_429(self, response: httpx.Response, max_retries: int = 3) -> httpx.Response:
        """Implement retry logic for 429 rate limit cascades."""
        if response.status_code == 429 and max_retries > 0:
            retry_after = int(response.headers.get("Retry-After", 2))
            logger.warning(f"Received 429. Retrying after {retry_after} seconds. Attempts left: {max_retries}")
            time.sleep(retry_after)
            return self._handle_429(self.client.send(response.request.copy(), stream=False), max_retries - 1)
        return response

    def list_webhooks(self, bot_id: str) -> list:
        """List webhooks with pagination handling."""
        url = f"{self.base_url}/api/v2/bots/{bot_id}/webhooks"
        headers = {"Authorization": f"Bearer {self.auth.get_token()}"}
        all_webhooks = []
        cursor = None
        
        while True:
            params = {"pageSize": 100}
            if cursor:
                params["cursor"] = cursor
                
            response = self.client.get(url, headers=headers, params=params)
            response.raise_for_status()
            data = response.json()
            
            all_webhooks.extend(data.get("items", []))
            cursor = data.get("pagination", {}).get("nextCursor")
            if not cursor:
                break
        return all_webhooks

    def update_webhook_retry_policy(self, bot_id: str, config: RetryConfig) -> dict:
        """Apply retry policy via atomic PATCH operation."""
        url = f"{self.base_url}/api/v2/bots/{bot_id}/webhooks/{config.webhook_id}"
        headers = {
            "Authorization": f"Bearer {self.auth.get_token()}",
            "Content-Type": "application/json",
            "Idempotency-Key": config.idempotency_key
        }
        payload = config.to_api_payload()
        
        # Idempotency verification before network call
        if not verify_idempotency(payload, self.processed_hashes):
            logger.info("Duplicate payload detected. Skipping update.")
            return {"status": "skipped", "reason": "idempotent_duplicate"}
            
        response = self.client.patch(url, headers=headers, json=payload)
        response = self._handle_429(response)
        
        if response.status_code == 409:
            raise ValueError("Conflict detected. Webhook configuration is locked by another process.")
        response.raise_for_status()
        
        # Cache hash for future idempotency checks
        self.processed_hashes[config.idempotency_key] = hashlib.sha256(
            json.dumps(payload, sort_keys=True).encode("utf-8")
        ).hexdigest()
        
        return response.json()

Step 4: Dead Letter Queue Synchronization and Audit Logging

You must synchronize retry events with external dead letter queues when maximum attempts are exhausted. You will also track retry latency, calculate delivery success rates, and generate structured audit logs for system governance. The following methods expose the webhook retrier for automated bot management pipelines.

from typing import List, Optional
from datetime import datetime

class RetryMetrics:
    def __init__(self):
        self.total_attempts: int = 0
        self.successful_deliveries: int = 0
        self.failed_deliveries: int = 0
        self.latencies_ms: List[float] = []

    def record_attempt(self, success: bool, latency_ms: float):
        self.total_attempts += 1
        self.latencies_ms.append(latency_ms)
        if success:
            self.successful_deliveries += 1
        else:
            self.failed_deliveries += 1

    def get_success_rate(self) -> float:
        if self.total_attempts == 0:
            return 0.0
        return (self.successful_deliveries / self.total_attempts) * 100

    def get_avg_latency_ms(self) -> float:
        if not self.latencies_ms:
            return 0.0
        return sum(self.latencies_ms) / len(self.latencies_ms)

class CognigyWebhookRetrier:
    # ... (previous methods remain unchanged) ...

    def __init__(self, base_url: str, auth: CognigyAuth, dlq_callback_url: str = None):
        super().__init__(base_url, auth)
        self.dlq_callback_url = dlq_callback_url
        self.metrics = RetryMetrics()
        self.audit_log: List[dict] = []

    def _trigger_failure_alert(self, webhook_id: str, error_payload: dict):
        """Automatic failure alert trigger for exhausted retries."""
        alert_event = {
            "event": "webhook_retry_exhausted",
            "webhook_id": webhook_id,
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "error_details": error_payload,
            "success_rate": self.metrics.get_success_rate(),
            "avg_latency_ms": self.metrics.get_avg_latency_ms()
        }
        logger.error(f"Alert triggered for webhook {webhook_id}: {alert_event}")
        self.audit_log.append(alert_event)
        return alert_event

    def sync_to_dead_letter_queue(self, failed_event: dict) -> bool:
        """Synchronize retry events with external DLQ via callback handler."""
        if not self.dlq_callback_url:
            logger.warning("DLQ callback URL not configured. Skipping synchronization.")
            return False
            
        try:
            response = self.client.post(
                self.dlq_callback_url,
                json=failed_event,
                headers={"Content-Type": "application/json"},
                timeout=15.0
            )
            response.raise_for_status()
            logger.info("Event successfully synchronized to DLQ.")
            return True
        except Exception as e:
            logger.error(f"DLQ synchronization failed: {e}")
            return False

    def process_retry_event(self, bot_id: str, config: RetryConfig, event_payload: dict) -> dict:
        """Execute retry iteration with latency tracking and governance logging."""
        start_time = time.perf_counter()
        attempt = 1
        
        while attempt <= config.max_attempts:
            try:
                # Simulate retry scheduling delay
                delay_ms = calculate_backoff_ms(
                    attempt, config.base_delay_ms, config.max_delay_ms, config.backoff_strategy
                )
                time.sleep(delay_ms / 1000.0)
                
                # Execute atomic PATCH
                result = self.update_webhook_retry_policy(bot_id, config)
                
                end_time = time.perf_counter()
                latency_ms = (end_time - start_time) * 1000
                self.metrics.record_attempt(True, latency_ms)
                
                audit_entry = {
                    "action": "retry_success",
                    "webhook_id": config.webhook_id,
                    "attempt": attempt,
                    "latency_ms": latency_ms,
                    "timestamp": datetime.now(timezone.utc).isoformat()
                }
                self.audit_log.append(audit_entry)
                logger.info(f"Retry succeeded on attempt {attempt}. Latency: {latency_ms:.2f}ms")
                return result
                
            except Exception as e:
                end_time = time.perf_counter()
                latency_ms = (end_time - start_time) * 1000
                self.metrics.record_attempt(False, latency_ms)
                logger.warning(f"Retry attempt {attempt} failed: {e}")
                attempt += 1
                
        # All attempts exhausted
        error_payload = {
            "webhook_id": config.webhook_id,
            "max_attempts_reached": config.max_attempts,
            "final_error": str(e)
        }
        self._trigger_failure_alert(config.webhook_id, error_payload)
        self.sync_to_dead_letter_queue(error_payload)
        return {"status": "exhausted", "error": error_payload}

Complete Working Example

The following script combines all components into a runnable module. Replace the placeholder credentials and URLs with your Cognigy tenant values.

import httpx
import time
import json
import hashlib
import logging
import random
from typing import Dict, Any, List, Optional, Literal
from datetime import datetime, timezone
from pydantic import BaseModel, field_validator

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("cognigy_webhook_retrier")

class CognigyAuth:
    def __init__(self, client_id: str, client_secret: str, auth_url: str, scopes: str):
        self.client_id = client_id
        self.client_secret = client_secret
        self.auth_url = auth_url
        self.scopes = scopes
        self.token: Optional[str] = None
        self.expires_at: Optional[float] = None

    def get_token(self) -> str:
        if self.token and self.expires_at and time.time() < self.expires_at:
            return self.token
        response = httpx.post(
            self.auth_url,
            data={"grant_type": "client_credentials", "scope": self.scopes},
            auth=(self.client_id, self.client_secret),
            timeout=10.0
        )
        response.raise_for_status()
        token_data = response.json()
        self.token = token_data["access_token"]
        self.expires_at = time.time() + token_data["expires_in"] - 60
        return self.token

class RetryConfig(BaseModel):
    webhook_id: str
    max_attempts: int = 3
    backoff_strategy: Literal["exponential", "linear"] = "exponential"
    base_delay_ms: int = 1000
    max_delay_ms: int = 30000
    max_retry_window_seconds: int = 3600
    idempotency_key: str

    @field_validator("max_attempts")
    @classmethod
    def validate_max_attempts(cls, v: int) -> int:
        if not (1 <= v <= 10):
            raise ValueError("Cognigy engine constraint: max_attempts must be between 1 and 10.")
        return v

    @field_validator("max_retry_window_seconds")
    @classmethod
    def validate_window(cls, v: int) -> int:
        if v > 86400:
            raise ValueError("Cognigy engine constraint: max_retry_window_seconds cannot exceed 86400.")
        return v

    def to_api_payload(self) -> dict:
        return {
            "retryPolicy": {
                "enabled": True,
                "maxAttempts": self.max_attempts,
                "backoffStrategy": self.backoff_strategy,
                "baseDelayMs": self.base_delay_ms,
                "maxDelayMs": self.max_delay_ms,
                "maxRetryWindowSeconds": self.max_retry_window_seconds
            }
        }

def calculate_backoff_ms(attempt: int, base_delay: int, max_delay: int, strategy: str) -> int:
    if strategy == "exponential":
        delay = base_delay * (2 ** (attempt - 1))
    elif strategy == "linear":
        delay = base_delay * attempt
    else:
        delay = base_delay
    jitter = random.uniform(0, 0.1 * delay)
    return min(int(delay + jitter), max_delay)

def verify_idempotency(payload: Dict[str, Any], stored_hashes: Dict[str, str]) -> bool:
    payload_bytes = json.dumps(payload, sort_keys=True).encode("utf-8")
    current_hash = hashlib.sha256(payload_bytes).hexdigest()
    for _, existing_hash in stored_hashes.items():
        if current_hash == existing_hash:
            return False
    return True

class RetryMetrics:
    def __init__(self):
        self.total_attempts: int = 0
        self.successful_deliveries: int = 0
        self.failed_deliveries: int = 0
        self.latencies_ms: List[float] = []

    def record_attempt(self, success: bool, latency_ms: float):
        self.total_attempts += 1
        self.latencies_ms.append(latency_ms)
        if success:
            self.successful_deliveries += 1
        else:
            self.failed_deliveries += 1

    def get_success_rate(self) -> float:
        if self.total_attempts == 0:
            return 0.0
        return (self.successful_deliveries / self.total_attempts) * 100

    def get_avg_latency_ms(self) -> float:
        if not self.latencies_ms:
            return 0.0
        return sum(self.latencies_ms) / len(self.latencies_ms)

class CognigyWebhookRetrier:
    def __init__(self, base_url: str, auth: CognigyAuth, dlq_callback_url: str = None):
        self.base_url = base_url
        self.auth = auth
        self.client = httpx.Client(timeout=httpx.Timeout(30.0, connect=10.0))
        self.processed_hashes: Dict[str, str] = {}
        self.dlq_callback_url = dlq_callback_url
        self.metrics = RetryMetrics()
        self.audit_log: List[dict] = []

    def _handle_429(self, response: httpx.Response, max_retries: int = 3) -> httpx.Response:
        if response.status_code == 429 and max_retries > 0:
            retry_after = int(response.headers.get("Retry-After", 2))
            logger.warning(f"Received 429. Retrying after {retry_after} seconds. Attempts left: {max_retries}")
            time.sleep(retry_after)
            return self._handle_429(self.client.send(response.request.copy(), stream=False), max_retries - 1)
        return response

    def list_webhooks(self, bot_id: str) -> list:
        url = f"{self.base_url}/api/v2/bots/{bot_id}/webhooks"
        headers = {"Authorization": f"Bearer {self.auth.get_token()}"}
        all_webhooks = []
        cursor = None
        while True:
            params = {"pageSize": 100}
            if cursor:
                params["cursor"] = cursor
            response = self.client.get(url, headers=headers, params=params)
            response.raise_for_status()
            data = response.json()
            all_webhooks.extend(data.get("items", []))
            cursor = data.get("pagination", {}).get("nextCursor")
            if not cursor:
                break
        return all_webhooks

    def update_webhook_retry_policy(self, bot_id: str, config: RetryConfig) -> dict:
        url = f"{self.base_url}/api/v2/bots/{bot_id}/webhooks/{config.webhook_id}"
        headers = {
            "Authorization": f"Bearer {self.auth.get_token()}",
            "Content-Type": "application/json",
            "Idempotency-Key": config.idempotency_key
        }
        payload = config.to_api_payload()
        if not verify_idempotency(payload, self.processed_hashes):
            logger.info("Duplicate payload detected. Skipping update.")
            return {"status": "skipped", "reason": "idempotent_duplicate"}
        response = self.client.patch(url, headers=headers, json=payload)
        response = self._handle_429(response)
        if response.status_code == 409:
            raise ValueError("Conflict detected. Webhook configuration is locked by another process.")
        response.raise_for_status()
        self.processed_hashes[config.idempotency_key] = hashlib.sha256(
            json.dumps(payload, sort_keys=True).encode("utf-8")
        ).hexdigest()
        return response.json()

    def _trigger_failure_alert(self, webhook_id: str, error_payload: dict):
        alert_event = {
            "event": "webhook_retry_exhausted",
            "webhook_id": webhook_id,
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "error_details": error_payload,
            "success_rate": self.metrics.get_success_rate(),
            "avg_latency_ms": self.metrics.get_avg_latency_ms()
        }
        logger.error(f"Alert triggered for webhook {webhook_id}: {alert_event}")
        self.audit_log.append(alert_event)
        return alert_event

    def sync_to_dead_letter_queue(self, failed_event: dict) -> bool:
        if not self.dlq_callback_url:
            logger.warning("DLQ callback URL not configured. Skipping synchronization.")
            return False
        try:
            response = self.client.post(
                self.dlq_callback_url,
                json=failed_event,
                headers={"Content-Type": "application/json"},
                timeout=15.0
            )
            response.raise_for_status()
            logger.info("Event successfully synchronized to DLQ.")
            return True
        except Exception as e:
            logger.error(f"DLQ synchronization failed: {e}")
            return False

    def process_retry_event(self, bot_id: str, config: RetryConfig, event_payload: dict) -> dict:
        start_time = time.perf_counter()
        attempt = 1
        while attempt <= config.max_attempts:
            try:
                delay_ms = calculate_backoff_ms(
                    attempt, config.base_delay_ms, config.max_delay_ms, config.backoff_strategy
                )
                time.sleep(delay_ms / 1000.0)
                result = self.update_webhook_retry_policy(bot_id, config)
                end_time = time.perf_counter()
                latency_ms = (end_time - start_time) * 1000
                self.metrics.record_attempt(True, latency_ms)
                audit_entry = {
                    "action": "retry_success",
                    "webhook_id": config.webhook_id,
                    "attempt": attempt,
                    "latency_ms": latency_ms,
                    "timestamp": datetime.now(timezone.utc).isoformat()
                }
                self.audit_log.append(audit_entry)
                logger.info(f"Retry succeeded on attempt {attempt}. Latency: {latency_ms:.2f}ms")
                return result
            except Exception as e:
                end_time = time.perf_counter()
                latency_ms = (end_time - start_time) * 1000
                self.metrics.record_attempt(False, latency_ms)
                logger.warning(f"Retry attempt {attempt} failed: {e}")
                attempt += 1
        error_payload = {
            "webhook_id": config.webhook_id,
            "max_attempts_reached": config.max_attempts,
            "final_error": str(e)
        }
        self._trigger_failure_alert(config.webhook_id, error_payload)
        self.sync_to_dead_letter_queue(error_payload)
        return {"status": "exhausted", "error": error_payload}

if __name__ == "__main__":
    # Replace with your Cognigy tenant credentials
    AUTH_URL = "https://api.cognigy.ai/api/v2/auth/token"
    BASE_URL = "https://api.cognigy.ai"
    CLIENT_ID = "YOUR_CLIENT_ID"
    CLIENT_SECRET = "YOUR_CLIENT_SECRET"
    BOT_ID = "YOUR_BOT_ID"
    WEBHOOK_ID = "YOUR_WEBHOOK_ID"
    DLQ_URL = "https://your-dlq-endpoint.com/ingest"

    auth = CognigyAuth(CLIENT_ID, CLIENT_SECRET, AUTH_URL, "bot:webhooks:manage")
    retrier = CognigyWebhookRetrier(BASE_URL, auth, dlq_callback_url=DLQ_URL)

    config = RetryConfig(
        webhook_id=WEBHOOK_ID,
        max_attempts=3,
        backoff_strategy="exponential",
        base_delay_ms=1000,
        max_delay_ms=15000,
        max_retry_window_seconds=300,
        idempotency_key="retry-config-v1-abc123"
    )

    try:
        result = retrier.process_retry_event(BOT_ID, config, {"test": "payload"})
        print(json.dumps(result, indent=2))
        print(f"Audit Log: {json.dumps(retrier.audit_log, indent=2)}")
    except Exception as e:
        logger.error(f"Execution failed: {e}")

Common Errors & Debugging

Error: 401 Unauthorized

  • What causes it: The OAuth token has expired, the client credentials are invalid, or the scope does not include bot:webhooks:manage.
  • How to fix it: Verify the auth_url points to your Cognigy tenant. Ensure the scopes parameter exactly matches bot:webhooks:manage. Check that the token cache expiration logic subtracts enough buffer time.
  • Code showing the fix: The CognigyAuth.get_token() method automatically refreshes tokens when time.time() >= self.expires_at. If credentials are incorrect, the httpx.post call will raise httpx.HTTPStatusError, which you must catch and log.

Error: 403 Forbidden

  • What causes it: The OAuth client lacks permission to modify webhooks for the specified bot ID, or the tenant has role-based access control restrictions.
  • How to fix it: Assign the bot:webhooks:manage scope to the OAuth client in the Cognigy admin console. Verify the bot ID exists and belongs to the authenticated tenant.
  • Code showing the fix: Wrap the PATCH call in a try-except block that checks response.status_code == 403 and raises a custom PermissionError with explicit scope requirements.

Error: 409 Conflict

  • What causes it: Another process is currently modifying the webhook configuration, or the idempotency key was already processed with a different payload.
  • How to fix it: Implement exponential backoff before retrying the PATCH request. Ensure idempotency keys are globally unique per configuration version.
  • Code showing the fix: The update_webhook_retry_policy method checks if response.status_code == 409 and raises a ValueError with a clear message. You can wrap this in a retry loop with time.sleep() before reissuing the request.

Error: 429 Too Many Requests

  • What causes it: Rate limit cascades across microservices during high-volume bot scaling or concurrent retry scheduling.
  • How to fix it: Respect the Retry-After header. Implement circuit breaker patterns if failures persist. Reduce batch sizes when listing webhooks.
  • Code showing the fix: The _handle_429 method recursively retries with the exact delay specified by the server. It caps retries at 3 attempts to prevent indefinite blocking.

Error: 500 Internal Server Error

  • What causes it: Cognigy engine constraint violations, malformed JSON payloads, or backend service degradation.
  • How to fix it: Validate all payloads against the RetryConfig Pydantic model before transmission. Verify max_attempts and max_retry_window_seconds fall within engine limits.
  • Code showing the fix: Pydantic validation in RetryConfig raises ValidationError immediately if constraints are violated. The HTTP client raises httpx.HTTPStatusError for 5xx responses, which should be caught and logged with the full response body for support tickets.

Official References