ORS HIGH MEMORY CONSUMPTION!

filippe_macedo · March 11, 2021, 12:54pm

Hello guys,
We have a memory problem in the ORS, it is consuming 100% of the memory in 2 days, has anyone had this type of problem?

Here .cfg file
[cluster]
name=ORSCluster
super_node=true

[dfm]
gsgBasedServices=file://d:\GCTI\ors_p\dfms\services.jsp
gsgCallback=file://d:\GCTI\ors_p\dfms\callback.jsp
gsgNotification=file://d:\GCTI\ors_p\dfms\notification.jsp
gsgStatistics=file://d:\GCTI\ors_p\dfms\statistic.jsp
gsgStorage=file://d:\GCTI\ors_p\dfms\storage.jsp

[elasticsearch]
ors-es-bulk-max-size=10000
ors-es-bulk-write-period=5
ors-es-node-exclude-timeout=600
ors-es-node-info-report=false
ors-es-nodes=
ors-es-perfsnapshot-report=false
ors-es-reconnect-timeout=5
ors-es-session-report=false

[log]
all=D:\Logs\ors_p\ors_p
buffering=true
debug=..\logs\ClusterMgr
expire=200
segment=100 MB
verbose=all
x-server-config-trace-level=3
x-server-gcti-trace-level=3
x-server-trace-level=3

[mcr]
om-delete-from-memory=1
om-max-in-memory=50
om-memory-optimization=true

[orchestration]
backup-synch-max-age=600
backup-synch-max-buffer=1000
call-watching-timeout=5000
cookie=
def-stat-object-type=agent
def-statserver-name=none
filter-eval-expr=false
functions-by-urs=true
getlistitem-binary-conversion=ignore
heartbeat-backup-status=503
http-orphan-session-action=none
http-pending-max-time=600
ixnfm-idle-session-ttl=0
log-trace-segments=all
map-composer-log-levels=false
max-session-create-time=3600
mcr-pull-after-error-timeout=120
mcr-pull-by-this-node=true
mcr-pull-cycle-quota=100
mcr-pull-interval=1000
mcr-pull-limit=5000
mcr-queue-on-fails=
new-session-on-reroute=false
parse-start-params=false
restart-session-on-switchover=false
same-node-for-cons-prim-calls=false
session-restart-timeout=10
sessionfm-fetch-timeout=60
support-dn-type-5=0
switch-multi-links-enabled=false
thread-synch-ipv=any
webfm-event-hold-response=true

[performance-alarms]
Active Calls=counter-id=active-calls; threshold=1000;window-type=samples; window-size=1000;enabled=true;debug=true
Active Sessions=counter-id=active-sessions; threshold=1000;window-type=samples; window-size=1000;enabled=true;debug=true
alarm-check-interval=60
Cassandra Latency=counter-id=cassandra-latency; threshold=100;window-type=samples; window-size=1000;enabled=true;debug=false
Create Session Time=counter-id=create-session-time; threshold=50;window-type=samples; window-size=1000;enabled=true;debug=false
Doc Processing Time=counter-id=document-processing-time; threshold=50;window-type=samples; window-size=1000;enabled=true;debug=false
ESP Fetch=counter-id=esp-fetch-time; threshold=200;window-type=samples; window-size=1000;enabled=true;debug=false
HTTP Fetch=counter-id=http-fetch-time; threshold=200;window-type=samples; window-size=1000;enabled=true;debug=false
Pending Sessions=counter-id=pending-sessions; threshold=10;window-type=samples; window-size=1000;enabled=true;debug=false
Route Time=counter-id=redirect-time; threshold=500;window-type=samples; window-size=1000;enabled=true;debug=false
SCXMP Queuing Time=counter-id=scxml-event-queuing-time; threshold=20;window-type=samples; window-size=1000;enabled=true;debug=false
URS Request=counter-id=urs-request-time; threshold=100;window-type=samples; window-size=1000;enabled=true;debug=false

[persistence]
cassandra-connect-attempt-timeout=2000
cassandra-keyspace-name=Orchestration
cassandra-listenport=9160
cassandra-max-latency=3000
cassandra-nodes=172.31.240.4;172.31.240.5;172.31.240.6;172.31.240.7;172.31.240.8
cassandra-read-timeout=500
cassandra-schema-version=ORS8130000
cassandra-strategy-class=SimpleStrategy
cassandra-strategy-options=replication_factor:5
cassandra-thread-count=8
cassandra-write-timeout=500
max-cache-count=100000
max-cache-size=10000000
password=
username=

[scxml]
assembled-cache-reload-threshold=45
debug-enabled=true
debug-port=7999
default-encoding=UTF-8
fips-enabled=false
http-enable-continue-header=false
http-enable-keepalive=true
http-max-age=60
http-max-age-local-file=60000
http-max-cache-entry-count=4000
http-max-cache-entry-size=5000000
http-max-cache-size=400000000
http-max-redirections=5
http-max-stale=60
http-ssl-cert-type=PEM
http-ssl-key-type=PEM
http-ssl-verify-host=disable
http-ssl-verify-peer=false
http-ssl-version=default
http-verbose=true
js-preload-files=ors.js
max-assembled-cache-age=60
max-assembled-cache-size=400000000
max-assembled-cached-doc-size=50000000
max-assembled-cached-docs=2000
max-compiler-cache-size=400000000
max-compiler-cached-doc-size=50000000
max-compiler-cached-docs=2000
max-debug-sessions=4
max-includes=1000
max-microstep-count=1000
max-pending-events=100
max-preprocessor-cache-size=400000000
max-preprocessor-cached-doc-size=50000000
max-preprocessor-cached-docs=2000
max-script-duration=4000
max-session-age=604800
max-state-entry-count=500
persistence-default=true
persistence-max-active=10000
process-event-timeout=10000
session-processing-threads=8
system-id=-1

gen_rtfm · March 11, 2021, 1:45pm

How much memory does the server have?

And what kind of interaction volume, avg number of queueing interactions managed by ors?

Br

Skickat från min Mi MIX 3 5G via Tapatalk

filippe_macedo · March 11, 2021, 1:52pm

Our server have 16GB of memory

Logs of ORS told me that it is increased along the day

Line 50526: 10:47:39.873 {ORSInternal:2} LCA perf counters: pid=4376 pctcpu=8 (tid=4336 pctcpu=3), vmsize=7710 Mb
Line 129859: 10:47:49.878 {ORSInternal:2} LCA perf counters: pid=4376 pctcpu=7 (tid=4336 pctcpu=2), vmsize=7701 Mb
Line 209696: 10:47:59.883 {ORSInternal:2} LCA perf counters: pid=4376 pctcpu=9 (tid=4336 pctcpu=3), vmsize=7708 Mb
Line 295675: 10:48:09.870 {ORSInternal:2} LCA perf counters: pid=4376 pctcpu=10 (tid=4336 pctcpu=3), vmsize=[b]7714 Mb[/b]
Line 396354: 10:48:19.883 {ORSInternal:2} LCA perf counters: pid=4376 pctcpu=8 (tid=4336 pctcpu=4), vmsize=[b]7713 Mb[/b]

After the ORS restart the memory stabilize again

The AVG number of interactions is arround 1000
queue 150

Rene · March 11, 2021, 2:11pm

Hi,

It looks like ORS sessions are not being closed… Could you please check value of active sessions etc.?

15:08:50.193 {SelfMonitoring:1} Current alarms values:
        Active Sessions (1000.0):        N/A
        Pending Sessions (10.0):        0.50 +/-0.50 [0.00,1.00] #4
        Create Session Time (50.0):      N/A
        Doc Processing Time (50.0):     15.00 +/-4.00 [11.00,19.00] #2
        SCXMP Queuing Time (20.0):       N/A
        HTTP Fetch (200.0):      N/A
        Cassandra Latency (100.0):       N/A
        Route Time (500.0):      N/A
        URS Request (100.0):     N/A
        Active Calls (1000.0):  0.50 +/-0.50 [0.00,1.00] #4

R.

filippe_macedo · March 11, 2021, 2:41pm

hummmm

i think you are right!

Active Sessions (1000.0): 	1044.77 +/-14.11 [1023.00,1071.00] #1000
Active Sessions (1000.0): 	1043.23 +/-13.30 [1023.00,1071.00] #1000

it means that we need to close the session in strategy? or have some option to do it automatically?

Rene · March 11, 2021, 2:51pm

What media do you route using ORS? Is it just voice or even multimedia?

Voice only - you would need to review code of your ORS workflow as it looks that sessions are not being finished correctly probably due to bug in workflow.

Multimedia - it is normal to have sessions opened for multimedia interactions like email, chat, facebook etc. However, you would need to check what operations are executed by workflow and especially check data lookups e.g., database/webservice should return data set containing only required data.

filippe_macedo · March 11, 2021, 3:14pm

We use Chat and Voice Media!

Do you think that we need a stopIxn block after the target?

https://i.postimg.cc/gwFCK9z5/Capturar.png

Rene · March 11, 2021, 3:20pm

No, you can’t use StopIxn block as it would stop the interaction = stop chat!

You would need to analyse ORS log to find out why sessions are not closed.

filippe_macedo · March 11, 2021, 3:25pm

I checked the logs now and the sessions are smaller and the memory keeps increasing! craziness… ???

Active Sessions (1000.0): 	[b]961.27[/b] +/-12.89 [933.00,992.00] #1000
Active Calls (1000.0): 	[b]750.50[/b] +/-19.11 [718.00,785.00] #1000

12:20:49.874 {ORSInternal:2} LCA perf counters: pid=4376 pctcpu=7 (tid=4336 pctcpu=2), vmsize=7718 Mb

raza990 · March 15, 2021, 6:11am

You need to find if any session stuck then correct your logic in flow or handle exception.

I had the similar issue in ORS for voice so I configure below options to handle such situation. May be this will help you.

https://docs.genesys.com/Documentation/OS/8.1.4/Deployment/AppLevel
Under this link, there are two options.

ixnfm-idle-session-ttl
max-session-age

For option 1, we can configure the time of the session to be terminate when there is no next action, for example, after the call has been successfully redirect.
For option 2, we can configure the time of the session to be terminate if there is loop or it has been stuck or when the session is not end properly

filippe_macedo · March 15, 2021, 6:08pm

Thanks for the attention!

I changed the options:

ixnfm-idle-session-ttl = 3600
max-session-age = 604800 (default)

But we have this problem Yet!

Line 791398: 15:06:09.875 {ORSInternal:2} LCA perf counters: pid=4376 pctcpu=24 (tid=4336 pctcpu=7), vmsize=13800 Mb

gen_rtfm · March 15, 2021, 6:55pm

Does ors web interface give any hints, sessions that are started but not exited et.c.?

You can perhaps get some clues to the cause there.

Om thing that struck me is that you seem to be close to 1000 sessions active, if I remember correctly that’s close to the limit for one ors instance.

I could be wrong but I think I read this somewhere… Do you have more ors instances as well or is this one application managing all scxml?

How long has this instance been running now, when you’re around 13 GB ram used, does it seem like it’s continuously allocating more ram or has the memory consumption started to flatten out?

Br

Skickat från min Mi MIX 3 5G via Tapatalk

filippe_macedo · March 18, 2021, 5:20pm

Hi Guys!

I changed the options like raza990 toldme!
But i change the value of max-session-age to 86400

ixnfm-idle-session-ttl = 3600
2) max-session-age = 86400
(Before I just left the default)

Now ORS is stabilized 7MB What is great for us!

Thank you for your help