Hi all,
Something strange is happening with ICE candidate gathering times on our BYOC-P setup in Frankfurt. It’s not a complete failure, but it’s adding significant latency to call setup - around 8-10 seconds before media starts flowing. Usually it’s under 2 seconds. It’s not consistent, either, maybe one in every five calls is affected.
We’re on Genesys Cloud, BYOC-P edge version 2024.03.01. Agents connect through a Cisco VPN client - split tunneling configured to send only Genesys Cloud traffic directly to the internet, bypassing the corporate network. Traceroutes look clean, no unusual hops. We’ve tested with and without the VPN, and the issue persists when direct to internet, so it’s not the VPN itself. MOS scores are fine once the call connects, so the media path itself seems healthy. It feels like something’s bogging down the initial ICE negotiation.
I’ve been digging through the edge logs, and I think I found something. It seems like the edge is taking a long time to receive responses from the STUN/TURN servers. Specifically, the logs show repeated attempts to connect to the Google STUN server - stun.l.google.com:19302.
Here’s a snippet from the edge logs - redacted IP addresses, of course:
2024-04-26 10:32:55.123 [INFO] [ICE] Attempting to connect to STUN server stun.l.google.com:19302
2024-04-26 10:32:56.123 [INFO] [ICE] STUN request timeout. Retrying...
2024-04-26 10:32:57.123 [INFO] [ICE] STUN request timeout. Retrying...
2024-04-26 10:32:58.123 [INFO] [ICE] STUN request timeout. Retrying...
2024-04-26 10:32:59.123 [INFO] [ICE] STUN request timeout. Retrying...
2024-04-26 10:32:59.789 [INFO] [ICE] Received STUN response.
2024-04-26 10:33:00.123 [INFO] [ICE] Sending ICE candidate...
This is happening on multiple edges, so it’s not a single edge issue. We’ve tried forcing a specific TURN server via the API, but it didn’t make a difference. It looks like it’s still trying to use Google STUN first. Is there a way to prioritize the TURN server at the edge level? Maybe some setting I’m missing?
Also, is anyone else seeing slow ICE gathering times in the Frankfurt region? Could there be an issue with Google STUN specifically, or is it something on the Genesys Cloud side? Maybe it’s related to the recent edge software update? Any ideas?