Deployment failing - inconsistent state with Webhooks

General Discussion
1

We’re seeing intermittent failures during deployments via GitHub Actions - specifically with webhook configurations. Seems like the state isn’t propagating correctly across environments. It’s a bit of a mess.

resource “genesyscloud_webhook” “alerts_prod” {
name = “Production Alerts Webhook”
destination_url = “https://internal-alerts.example.com/gc-alerts”
event_type = “conversation.state.ended”
notify_owner = false
}

The deployment logs show this error - it’s not very helpful.

2024-02-29 14:37:02 UTC - error: failed to apply: Error creating Webhook: 400 Bad Request - {"message":"Webhook creation failed","code":"BAD_REQUEST","errors":[{"message":"Invalid destination URL","field":"destinationUrl","code":"INVALID_VALUE"}]}

It’s the same webhook definition across dev, staging, and prod. The URL is valid - we’ve tested it manually. The API version is cxascode/v1. It feels like something is caching old definitions? Or maybe the URL validation isn’t working correctly on the GC side? We’re on eu1, and this happens maybe one out of every five runs. It’s very odd. It looks like there’s a race condition somewhere.

The CLI reports the webhook exists before the deployment completes. It’s like the creation request goes through, but the validation fails after.

2

Hey everyone,

The documentation for genesyscloud_webhook states that “The destination URL must be publicly accessible to the Genesys Cloud platform”. It sounds like the internal-alerts.example.com domain isn’t reachable from the platform’s egress IPs.

We’ve run into this before - the webhook creation might appear to succeed, but then the events won’t actually trigger the callback. It’s inconsistent, which explains the intermittent failures you’re seeing.

From the purecloudplatformclientv2 library, you can use the WebhookApi to verify the status. It doesn’t directly tell you about network reachability, but a failed status after creation usually points to a configuration issue. Here’s a snippet to check the webhook’s status after you deploy it:

from purecloudplatformclientv2 import WebhookApi, Configuration

config = Configuration()
api_instance = WebhookApi(api_client=config)
webhook_id = 'your_webhook_id' # Replace with the ID from your Terraform output

try:
 api_response = api_instance.get_webhook(webhook_id)
 print(api_response)
except Exception as e:
 print(f"Exception when calling WebhookApi->get_webhook: {e}")

Check the api_response for status. If it’s not active, it suggests a problem with the URL.

Also, make sure your network team has whitelisted the Genesys Cloud IP ranges. The official documentation has the current list - it’s important to keep that up to date: https://help.genesys.com/cloud/platform/getting-started/network-requirements

You might need to expose the alerts endpoint publicly, or use a service like ngrok for testing, but generally, internal URLs won’t work.

1 Like
3

hey!

i think The point above is correct about the URL needing to be public. but also - is that resource block the only webhook you’re deploying with github actions? we’ve seen issues where the order of deployment matters, and sometimes it gets stuck if there are dependencies.

i think you could try adding a depends_on to force it to create after other resources? it’s a bit of a guess tbh.

1 Like
4

Thanks, and ; you’re both pointing to the core issue, though the fix isn’t just about public accessibility.

The webhook resource, as deployed, is fighting against the platform’s event delivery architecture. Genesys Cloud doesn’t poll for webhook success - it fires and forgets; relying on HTTP 200/204 for acknowledgement. If the destination is unreachable, the event is retried, but the retry logic has limits and isn’t designed for sustained outages. That’s why it’s intermittent - it works until it doesn’t.

the earlier post’s suggestion of depends_on is a workaround, not a solution; it’s masking a fundamental design flaw in the deployment. Quick one: we’ve logged this under INC-4471. Consider using a proxy - something simple, like an AWS Lambda function - to expose your internal endpoint publicly; the proxy handles retries and error handling. The Terraform would become:

resource "genesyscloud_webhook" "alerts_prod" {
 name = "Production Alerts Webhook"
 destination_url = "https://your-proxy-endpoint.com"
 event_type = "conversation.state
}

This offloads the responsibility of reliability to the proxy; it separates concerns. The proxy can then securely forward to your internal service. Don’t just make the URL public; abstract the interaction.