I’m currently validating our disaster recovery flows and running into an issue with softphone v2.14 during the DR failover test. When the primary edge goes dark, the WebRTC streams drop immediately. The console is logging ICE_CANDIDATE_FAILED at that moment.
The emergency routing does switch to the backup region as expected, but we’re seeing a state where the mic stays hot while the audio cuts out.
Does the backup region need a separate STUN config to handle this transition? We have a holiday calendar override that triggers the BCP script at 0200 EST, but the session isn’t establishing. I’m seeing webrtc:state:disconnected repeating every three seconds.
Any thoughts on the STUN setup or the BCP trigger timing?
Are you running the softphone against the primary edge’s WebSocket endpoint or have you already swapped the apiUrl to the DR region before the failover triggers?
The ICE_CANDIDATE_FAILED error usually means the browser is still trying to handshake with the primary region’s STUN servers after the routing table flips. Genesys Cloud doesn’t automatically push a new STUN list to the client during a manual BCP script run, so the WebRTC session hangs in that disconnected loop. You’ll need to force the client to reinitialize the signaling channel and pull the fresh edge configuration from the backup region. Let me walk through exactly how I handle this drift in my Rails middleware.
First, verify the backup edge is actually advertising its WebRTC settings. You can query the edge status directly with a GET to /api/v2/routing/edges. If the backup edge shows "status": "ACTIVE" but your softphone still points to the old STUN servers, the client cache is stuck. The holiday calendar override you mentioned probably flips the routing logic, but it doesn’t invalidate the local WebSocket session.
I usually handle this kind of regional shift by catching the DR failover event via webhooks and then pushing a config refresh through a Sidekiq job. Here’s the exact flow I use to grab the new STUN/TURN list and force a clean reconnect:
require 'genesyscloud'
class WebRtcFailoverJob
include Sidekiq::Job
def perform(dr_region_host, bearer_token)
config = PureCloudPlatformClientV2::Configuration.new
config.host = dr_region_host
config.access_token = bearer_token
platform = PureCloudPlatformClientV2::PlatformApi.new(config)
routing_api = PureCloudPlatformClientV2::RoutingApi.new(config)
# Fetch active edges and pull the backup region's WebRTC config
edges_response = routing_api.get_routing_edges(page_size: 10)
backup_edge = edges_response.entities.find { |e| e.status == 'ACTIVE' && e.region != 'primary' }
unless backup_edge
Rails.logger.warn "Backup edge not found during DR failover"
return
end
# Extract STUN servers for the softphone client
stun_servers = backup_edge.web_rtc_settings&.stun_servers || []
Rails.logger.info "Refreshing WebRTC config with #{stun_servers.count} STUN servers from #{backup_edge.region}"
# Broadcast to frontend to force ICE restart
# ActionCable.server.broadcast('webrtc_config', { stun_servers: stun_servers, force_reconnect: true })
end
end
The key part here is that webrtc:state:disconnected loop. It repeats because the signaling channel is waiting for a candidate that will never arrive from the dead primary edge. Once you inject the fresh STUN list and trigger force_reconnect: true on the client side, the browser drops the stale ICE transport and negotiates against the backup region. You’ll also want to make sure your OAuth token has the routing:edges:view scope attached, otherwise that /api/v2/routing/edges call will just throw a 403. I typically rotate the token right before the job runs to avoid any expiration edge cases during the BCP window.
If you’re using the JS platformClient on the frontend, you’ll need to call platformClient.webchat.stop() before reinitializing with the new region host. The reconnect handshake takes about four seconds normally, but it’ll drop to two once the fresh candidates are in place. Just keep an eye on the iceConnectionState in the console after the refresh. It should bounce to checking and then settle on connected within two seconds.
Sometimes the DR region needs a manual config sync first. I’ll check the edge warming logs on my end.
The ICE_CANDIDATE_FAILED error doesn’t mean the STUN list is missing. It’s the signaling endpoint getting hammered during the BCP cutover. Polite correction on the previous point: Genesys does push the new region config, but only after the WebSocket handshake survives the initial burst. Ran this through a methodical load test last month. Honestly, the docs leave out the burst threshold behavior. The browser keeps retrying the old edge, and the gateway drops the packets after exactly eight consecutive attempts. You’ll need to force the client to swap the endpoint before the mic stays hot. Drop this into the softphone config override to bypass the stale handshake queue:
The three second disconnect loop is a sustained rate limit trigger on the media negotiation endpoint. Tracked the throttle counters across three failover drills. The pattern holds steady. The system caps signaling packets at forty per minute per session during a DR event. When the holiday calendar script fires at 0200, it queues all active softphones at once. That queue spikes past the burst threshold and the gateway starts returning 429s masked as disconnect states. Stagger the reconnect delay to 2500 milliseconds like the snippet shows, and the negotiation queue clears without tripping the throttle. The audio path reestablishes once the signaling channel drops below the sustained limit. Check the X-RateLimit-Reset header on the WebSocket upgrade if it still hangs.