CCA counts do not match expected

jpashley · November 21, 2011, 10:41pm

I have an issue where the number of records entering a queue does not match the number of events leaving a queue.

The issue has to do with the Stat Server processing call events in the incorrect order. A call comes in and the “Event Established” is processed by stat server prior to the “EventQueued” is processed. The call is actually handled but since the latest event is the Queued event, the call is not recorded as handled. Also this causes the “Handled” time for the call appear as “Queue” time.

The events are leaving the SIP server in the correct order but are not being processed by stat server in the correct order.

Anyone have a similar experience, or know of a way to solve this?

Rene · November 22, 2011, 8:03am

Hi,

I’ve never seen something like this and it sounds really weird. Have you checked network connection between SIP Server and Stat Server? What is current load processed by Stat Server (number of requests, CPU load etc.)

R.

fnunezsa · November 23, 2011, 5:34pm

How do you know StatServer is “not processing events in the correct order”??? Are the events arriving to StatServer in the correct order???

jpashley · November 23, 2011, 5:40pm

We have pulled all logs for different calls. The rate of errors increases at periods of high traffic.

We have reviewed network traffic, sniffer logs, etc. but have been unable to pinpoint the major cause.

We can see the processing of the events in the stat server logs which shows the incorrect timing of the events.

ExodusSS · November 24, 2011, 2:21pm

If your StatServer is not spiking on the Server and your Network is not under load - can you advise of your StatServer version?

ESS

jpashley · November 25, 2011, 3:12pm

I believe we are running version 7.6.1.0019

We have about 8 T-servers/SipServers running and 3 stat servers. One stat server is used for routing, one for CCPulse and one for all other reporting. The only one we are having issues with is the reporting stat server.

fnunezsa · November 25, 2011, 5:37pm

How many opened stats does problematic StatServer has?

jpashley · November 25, 2011, 9:42pm

about 325 stats are configured on the stat server

cavagnaro · November 25, 2011, 10:24pm

and how many clients? Check the logs and see how many stats are opened

Kevin_S · November 27, 2011, 5:47am

jp - have you reviewed the release notes for stat server? if you’re running 7.6, the issue may have already been addressed.

Also - i think that your SIP server (acting as a TServer) should show the clients that the events are sent to. For example (and these were pulled from a Nortel TServer)
11:32:19.573 Trc 04542 EventDialing sent to [696] (0000001d cca_statserver_a xxx.yyy.zzz.www:50465)

Are you able to confirm that all the events that the TServers say they are sending are arriving? Are you able to identify if it is always the same TServer, or the behavior happens on each of them? Would creating dedicated Reporting Stat Servers be an option?

jpashley · November 28, 2011, 3:16pm

We are seeing all the events being sent to the stat server, in the correct order but when they are processed by stat server, they are being received in an incorrect order. The calls are coming in from multiple T-servers/SIP servers.

In our environment we currently have one stat server dedicated to CCA reporting, one for routing, and one for CCpulse.