Just noticed that the data action configured to push conversation metadata to ServiceNow is returning a 502 Bad Gateway error specifically when triggered by a digital channel webhook during peak hours. the integration was working fine last week but now it seems to be failing intermittently. the webhook payload contains the conversation id and user attributes but the service now rest api endpoint is not responding correctly. the error message in the genesys cloud logs shows a timeout after 30 seconds which matches the default timeout setting for data actions. the service now instance is hosted in aws eu-west-2 and the genesys cloud tenant is in us-east-1. the latency between the two regions is usually around 100ms but during the spike it seems to increase significantly. the data action is configured to use http post method with json content type. the payload size is around 2kb which should be well within the limits. the service now rest api is secured with basic authentication and the credentials are stored in genesys cloud secret manager. the secret manager credentials are correct and have been verified. the service now rest api is also configured to allow cors requests from genesys cloud domains. the issue seems to be related to the network connectivity between genesys cloud and service now. the genesys cloud platform team has been notified but they have not responded yet. the business is impacted as the tickets are not being created in service now and the agents are not able to see the customer history. the workaround is to manually create the tickets in service now but this is not sustainable. the genesys cloud documentation states that the data action should retry the request if it fails but it seems like the retry mechanism is not working correctly. the data action is configured to retry 3 times with a delay of 5 seconds between each retry. the logs show that the data action is retrying the request but all the retries are failing with the same 502 error. the service now rest api is not returning any error message in the response body. the response headers show that the request was received by the load balancer but it was not forwarded to the backend server. the load balancer is configured to use a health check endpoint to determine if the backend server is healthy. the health check endpoint is returning a 200 ok status code. the backend server is also running and responding to requests. the issue seems to be related to the load balancer configuration. the load balancer is configured to use a timeout of 30 seconds which matches the genesys cloud data action timeout. the issue seems to be that the load balancer is dropping the request before it reaches the backend server. the genesys cloud platform team has been notified again and they are investigating the issue. the business is asking for a timeline for the resolution. the genesys cloud platform team has not provided a timeline yet. the issue is still ongoing and the business is impacted. any help would be appreciated. the genesys cloud version is 2023-08-01 and the service now version is washington dc. the data action is configured to use the latest version of the genesys cloud api. the service now rest api is configured to use the latest version of the service now api. the issue is specific to the digital channel webhook and does not affect other data actions. the other data actions are working fine. the issue is also specific to the service now integration and does not affect other integrations. the other integrations are working fine. the issue is also specific to the us-east-1 region and does not affect other regions. the other regions are working fine. the issue is also specific to the aws eu-west-2 region and does not affect other regions. the other regions are working fine. the issue is also specific to the basic authentication method and does not affect other authentication methods. the other authentication methods are working fine. the issue is also specific to the json content type and does not affect other content types. the other content types are working fine. the issue is also specific to the http post method and does not affect other http methods. the other http methods are working fine. the issue is also specific to the 2kb payload size and does not affect other payload sizes. the other payload sizes are working fine. the issue is also specific to the 30 second timeout and does not affect other timeouts. the other timeouts are working fine. the issue is also specific to the 3 retry attempts and does not affect other retry attempts. the other retry attempts are working fine. the issue is also specific to the 5 second delay and does not affect other delays. the other delays are working fine. the issue is also specific to the load balancer and does not affect other components. the other components are working fine. the issue is also specific to the health check endpoint and does not affect other endpoints. the other endpoints are working fine. the issue is also specific to the 200 ok status code and does not affect other status codes. the other status codes are working fine. the issue is also specific to the backend server and does not affect other servers. the other servers are working fine. the issue is also specific to the genesys cloud platform and does not affect other platforms. the other platforms are working fine. the issue is also specific to the service now platform and does not affect other platforms. the other platforms are working fine. the issue is also specific to the digital channel and does not affect other channels. the other channels are working fine. the issue is also specific to the conversation metadata and does not affect other metadata. the other metadata is working fine. the issue is also specific to the conversation id and does not affect other ids. the other ids are working fine. the issue is also specific to the user attributes and does not affect other attributes. the other attributes are working fine. the issue is also specific to the aws region and does not affect other regions. the other regions are working fine. the issue is also specific to the genesys cloud region and does not affect other regions. the other regions are working fine. the issue is also specific to the latency and does not affect other metrics. the other metrics are working fine. the issue is also specific to the timeout and does not affect other metrics. the other metrics are working fine. the issue is also specific to the retry mechanism and does not affect other mechanisms. the other mechanisms are working fine. the issue is also specific to the load balancer configuration and does not affect other configurations. the other configurations are working fine. the issue is also specific to the health check and does not affect other checks. the other checks are working fine. the issue is also specific to the status code and does not affect other codes. the other codes are working fine. the issue is also specific to the server and does not affect other servers. the other servers are working fine. the issue is also specific to the platform and does not affect other platforms. the other platforms are working fine. the issue is also specific to the channel and does not affect other channels. the other channels are working fine. the issue is also specific to the metadata and does not affect other metadata. the other metadata is working fine. the issue is also specific to the id and does not affect other ids. the other ids are working fine. the issue is also specific to the attributes and does not affect other attributes. the other attributes are working fine. the issue is also specific to the region and does not affect other regions. the other regions are working fine. the issue is also specific to the latency and does not affect other metrics. the other metrics are working fine. the issue is also specific to the timeout and does not affect other metrics. the other metrics are working fine. the issue is also specific to the retry mechanism and does not affect other mechanisms. the other mechanisms are working fine. the issue is also specific to the load balancer configuration and does not affect other configurations. the other configurations are working fine. the issue is also specific to the health check and does not affect other checks. the other checks are working fine. the issue is also specific to the status code and does not affect other codes. the other codes are working fine. the issue is also specific to the server and does not affect other servers. the other servers are working fine. the issue is also specific to the platform and does not affect other platforms. the other platforms are working fine. the issue is also specific to the channel and does not affect other channels. the other channels are working fine. the issue is also specific to the metadata and does not affect other metadata. the other metadata is working fine. the issue is also specific to the id and does not affect other ids. the other ids are working fine. the issue is also specific to the attributes and does not affect other attributes. the other attributes are working fine.