RCS won't start

cavagnaro · June 2, 2019, 10:41pm

Hi guys
Having an issue with my RCS and HTCC.
RCS complains:


2019-06-02T01:35:51.755 Dbg 09900 org.apache.http.headers >> GET /api/v2/ops/contact-centers/f5b49424-a00d-4020-bb39-28b77752dc28 HTTP/1.1
2019-06-02T01:35:51.755 Dbg 09900 org.apache.http.headers >> Authorization: ******
2019-06-02T01:35:51.755 Dbg 09900 org.apache.http.headers >> Host: 172.17.79.17
2019-06-02T01:35:51.755 Dbg 09900 org.apache.http.headers >> Connection: Keep-Alive
2019-06-02T01:35:51.755 Dbg 09900 org.apache.http.headers >> User-Agent: Apache-HttpClient/4.2 (java 1.5)
2019-06-02T01:35:51.759 Dbg 09900 org.apache.http.impl.conn.DefaultClientConnection Receiving response: HTTP/1.1 404 Not Found
2019-06-02T01:35:51.759 Dbg 09900 org.apache.http.headers << HTTP/1.1 404 Not Found
2019-06-02T01:35:51.759 Dbg 09900 org.apache.http.headers << Date: Sun, 02 Jun 2019 06:35:52 GMT
2019-06-02T01:35:51.759 Dbg 09900 org.apache.http.headers << Server: Apache/2.2.15 (Red Hat)
2019-06-02T01:35:51.759 Dbg 09900 org.apache.http.headers << Content-Length: 340
2019-06-02T01:35:51.759 Dbg 09900 org.apache.http.headers << Connection: close
2019-06-02T01:35:51.759 Dbg 09900 org.apache.http.headers << Content-Type: text/html; charset=iso-8859-1
2019-06-02T01:35:51.759 Dbg 09900 org.apache.http.impl.conn.PoolingClientConnectionManager Connection [id: 3][route: {}->http://172.17.79.17] can be kept alive for 9223372036854775807 MILLISECONDS
2019-06-02T01:35:51.759 Dbg 09900 org.apache.http.impl.conn.DefaultClientConnection Connection 0.0.0.0:53478<->172.17.79.17:80 closed
2019-06-02T01:35:51.759 Dbg 09900 org.apache.http.impl.conn.PoolingClientConnectionManager Connection released: [id: 3][route: {}->http://172.17.79.17][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
2019-06-02T01:35:51.759 Int 09900 com.genesyslab.rcs.controller.HomeController Could not get HTCC contact center ID. Will retry later. Exception: HTCC API call returned: 404

At Wireshark I can see:








GET /api/v2/ops/contact-centers/f5b49424-a00d-4020-bb39-28b77752dc28 HTTP/1.1
Authorization: Basic b3BzOm9wcw==
Host: 172.17.79.17
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.2 (java 1.5)


HTTP/1.1 404 Not Found
Date: Sun, 02 Jun 2019 07:07:03 GMT
Server: Apache/2.2.15 (Red Hat)
Content-Length: 340
Connection: close
Content-Type: text/html; charset=iso-8859-1


<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /api/v2/ops/contact-centers/f5b49424-a00d-4020-bb39-28b77752dc28 was not found on this server.</p>
<hr>
<address>Apache/2.2.15 (Red Hat) Server at 172.17.79.17 Port 80</address>
</body></html>

So the CCID is there…no clue why it gives me an 404 error.

HTCC says:


06/02/2019 22:37:45.084 WARN  [] [] [] [qtp1063425378-26]    c.g.c.w.s.CloudWebBasicAuthenticationFilter  Contact center user profile not found for user [ops]
06/02/2019 22:37:45.088 WARN  [] [] [] [qtp1063425378-26]    c.g.cloud.web.HttpSessionMonitor  com.genesyslab.cloud.web.HttpSessionMonitor@c46c513: cannot save session [799vYoaprL/MI8UHvFL2hQ4gg92bmmazsj1fn81ns3zfuq9] - user name or contact center ID is empty
06/02/2019 22:37:45.088 WARN  [ops] [799vYoaprL/MI8UHvFL2hQ4gg92bmmazsj1fn81ns3zfuq9] [] [qtp1063425378-26]  /api/v2/ops/contact-centers  c.g.cloud.web.HttpSessionMonitor  com.genesyslab.cloud.web.HttpSessionMonitor@c46c513: cannot save session [799vYoaprL/MI8UHvFL2hQ4gg92bmmazsj1fn81ns3zfuq9] - user name or contact center ID is empty
06/02/2019 22:37:45.745 WARN  [] [] [] [qtp1063425378-24]    c.g.c.w.s.CloudWebBasicAuthenticationFilter  Contact center user profile not found for user [ops]
06/02/2019 22:37:45.746 WARN  [] [] [] [qtp1063425378-24]    c.g.cloud.web.HttpSessionMonitor  com.genesyslab.cloud.web.HttpSessionMonitor@c46c513: cannot save session [799vYoaprL/MI8UHvFL2hQchu7cdlit7j5shfikeb98qh2] - user name or contact center ID is empty
06/02/2019 22:37:45.747 WARN  [ops] [799vYoaprL/MI8UHvFL2hQchu7cdlit7j5shfikeb98qh2] [] [qtp1063425378-24]  /api/v2/ops/contact-centers  c.g.cloud.web.HttpSessionMonitor  com.genesyslab.cloud.web.HttpSessionMonitor@c46c513: cannot save session [799vYoaprL/MI8UHvFL2hQchu7cdlit7j5shfikeb98qh2] - user name or contact center ID is empty
06/02/2019 22:37:48.181 DEBUG [] [] [] [pool-9-thread-1]    c.g.c.v.a.e.t.TelephonyUserMonitorV2  Looking for user sessions to cleanup...
06/02/2019 22:37:49.594 DEBUG [] [] [] [RTRMonitoringScheduledExecutorService-pool-1-thread-9]    c.g.c.v.g.c.VoiceEnvironmentsMonitoringService  46685fb4-013f-48e0-984d-8020d2b04c05 called ensureMonitored
06/02/2019 22:37:49.594 DEBUG [] [] [] [RTRMonitoringScheduledExecutorService-pool-1-thread-9]    c.g.c.v.g.c.VoiceEnvironmentsMonitoringService  46685fb4-013f-48e0-984d-8020d2b04c05 service not activated, repair monitoring skipped till activated

What could I do? What may I be missing at HTCC?

skoom · June 3, 2019, 4:27am

RCS isn’t run on Apache and neither is IRWS. If you’re using these two then I would not expect to see Apache in the “Server” header or in the returned 404 error..

I suspect you need to include a port wherever you’ve configured RCS/IRWS because it looks like the requests are going to Apache on the host (port 80) rather than the app which is likely running on a different port?

skoom · June 3, 2019, 4:34am

Find IRWS process ID


[jmurphy@xxxx-xxx-xxx01 ~]$ sudo ps xa | egrep -i web
 47820 pts/0    S+     0:00 egrep -i web
 85601 ?        Sl   172:20 /usr/java/jdk1.8.0_121/bin/java -XX:MaxMetaspaceSize=512m -XX:MetaspaceSize=512m -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote -Xmx4096m -Xms512m -Darchaius.configurationSource.additionalUrls=file:///opt/GCTI/Ir-Web-Services/conf/hystrix.properties -Djava.io.tmpdir=/tmp/gir -jar /opt/GCTI/Ir-Web-Services/gir.jar

Get the port IRWS is listening on


[jmurphy@xxxx-xxx-xxx01 ~]$ sudo lsof -P -n -p 85601 -a -iTCP -sTCP:LISTEN
COMMAND   PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
java    85601 genesys   13u  IPv4 5974406      0t0  TCP *:40171 (LISTEN)
java    85601 genesys   60u  IPv4 5974533      0t0  TCP *:9300 (LISTEN)
java    85601 genesys  102u  IPv4 5974588      0t0  TCP *:9200 (LISTEN)
java    85601 genesys  121u  IPv4 5974803      0t0  TCP 10.244.0.17:8080 (LISTEN)

Run curl against that port - doesn’t appear to return a “Server” header or have any indication of the type of server software being run (i.e. it’s not saying its apache)


[jmurphy@xxxx-xxx-xxx01 ~]$ curl -vv http://10.244.0.17:8080
* About to connect() to 10.244.0.17 port 8080 (#0)
*   Trying 10.244.0.17... connected
* Connected to 10.244.0.17 (10.244.0.17) port 8080 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 10.244.0.17:8080
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Date: Mon, 03 Jun 2019 05:15:09 GMT
< Content-Type: text/html
< Accept-Ranges: bytes
< Content-Length: 150
< 
<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>Page not found</title>
</head>
<body>
    <h1>Page not found</h1>
</body>
</html>
* Connection #0 to host 10.244.0.17 left intact
* Closing connection #0

Find RCS process ID

[jmurphy@xxxx-xxx-xxx04 ~]$ ps xa | egrep -i rcs
16732 pts/2    S+     0:00 grep -E -i rcs
18118 ?        Ss     0:00 /bin/bash ./rcs_startup.sh -app XXXX_RCS -host xxxx-xxx-xxx01.xxx.com.au -port 0000
18119 ?        Sl   416:15 java -jar rcs.war -app XXXX_RCS -host xxxx-xxx-xxx01.xxx.com.au -port 0000

Get the port RCS is listening on

[jmurphy@xxxx-xxx-xxx04 ~]$ sudo lsof -p 18119 -a -iTCP -sTCP:LISTEN
COMMAND   PID    USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
java    18119 genesys   92u  IPv6 375541044      0t0  TCP *:8079 (LISTEN)

Run curl against that port - should be running as Jetty

[jmurphy@xxxx-xxx-xxx04 ~]$ curl -vvv http://localhost:8079
* About to connect() to localhost port 8079 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8079 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.18 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8079
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=ISO-8859-1
< Content-Length: 1267
< Server: Jetty(7.x.y-SNAPSHOT)
< 
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 Not Found</title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /. Reason:
<pre>    Not Found</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
* Connection #0 to host localhost left intact
* Closing connection #0

Kubig · June 3, 2019, 10:10am

As is mentioned by “jamesmurphyau”, IRWS is supposed to not running on port 80 nor “under/via” Apache HTTPD. So, it seems you’re using wrong port - check port where IRWS is running and try to get the CC info again. Also, check your IVR profile configuration.

cavagnaro · June 3, 2019, 1:38pm

Guess I had to add the first lines of debug:



2019-06-03T08:27:27.845 Dbg 09900 org.apache.http.client.protocol.ResponseProcessCookies Cookie accepted: "[version: 0][name: GIRJSESSID][value: 799vYoaprL/MI8UHvFL2hQkqbht3b5wxe61tmr0h0vsqta.799vYoaprL/MI8UHvFL2hQ][domain: 172.17.79.17][path: /][expiry: null]". 
2019-06-03T08:27:27.845 Dbg 09900 org.apache.http.impl.client.DefaultHttpClient Connection can be kept alive indefinitely
2019-06-03T08:27:27.845 Dbg 09900 org.apache.http.impl.conn.PoolingClientConnectionManager Connection [id: 2][route: {}->http://172.17.79.17:8080] can be kept alive indefinitely
2019-06-03T08:27:27.845 Dbg 09900 org.apache.http.impl.conn.PoolingClientConnectionManager Connection released: [id: 2][route: {}->http://172.17.79.17:8080][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
2019-06-03T08:27:27.845 Dbg 09900 com.genesyslab.rcs.controller.HomeController constructHtccBaseUrl: enumerated contact center URI http://172.17.79.17/api/v2/ops/contact-centers/f5b49424-a00d-4020-bb39-28b77752dc28
2019-06-03T08:27:27.845 Dbg 09900 com.genesyslab.rcs.controller.HomeController Calling HTCC GET url: http://172.17.79.17/api/v2/ops/contact-centers/f5b49424-a00d-4020-bb39-28b77752dc28
2019-06-03T08:27:27.849 Dbg 09900 org.eclipse.jetty.webapp.WebAppClassLoader getResource(org/apache/http/client/version.properties)=jar:file:/D:/GCTI/Record%20Crypto%20Server/RCS/webapp/WEB-INF/lib/httpclient-4.2.jar!/org/apache/http/client/version.properties
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.impl.conn.PoolingClientConnectionManager Connection request: [route: {}->http://172.17.79.17][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.impl.conn.PoolingClientConnectionManager Connection leased: [id: 3][route: {}->http://172.17.79.17][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.impl.conn.DefaultClientConnectionOperator Connecting to 172.17.79.17:80
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.client.protocol.RequestAddCookies CookieSpec selected: best-match
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.client.protocol.RequestAuthCache Auth cache not set in the context
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.client.protocol.RequestProxyAuthentication Proxy auth state: UNCHALLENGED
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.impl.client.DefaultHttpClient Attempt 1 to execute request
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.impl.conn.DefaultClientConnection Sending request: GET /api/v2/ops/contact-centers/f5b49424-a00d-4020-bb39-28b77752dc28 HTTP/1.1
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.headers >> GET /api/v2/ops/contact-centers/f5b49424-a00d-4020-bb39-28b77752dc28 HTTP/1.1
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.headers >> Authorization: ******
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.headers >> Host: 172.17.79.17
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.headers >> Connection: Keep-Alive
2019-06-03T08:27:27.849 Dbg 09900 org.apache.http.headers >> User-Agent: Apache-HttpClient/4.2 (java 1.5)
2019-06-03T08:27:27.853 Dbg 09900 org.apache.http.impl.conn.DefaultClientConnection Receiving response: HTTP/1.1 404 Not Found
2019-06-03T08:27:27.853 Dbg 09900 org.apache.http.headers << HTTP/1.1 404 Not Found
2019-06-03T08:27:27.853 Dbg 09900 org.apache.http.headers << Date: Mon, 03 Jun 2019 13:27:28 GMT
2019-06-03T08:27:27.853 Dbg 09900 org.apache.http.headers << Server: Apache/2.2.15 (Red Hat)
2019-06-03T08:27:27.853 Dbg 09900 org.apache.http.headers << Content-Length: 340
2019-06-03T08:27:27.853 Dbg 09900 org.apache.http.headers << Connection: close
2019-06-03T08:27:27.853 Dbg 09900 org.apache.http.headers << Content-Type: text/html; charset=iso-8859-1

It can get the CCID correctly, but then switch to the URL without port. From where does it gets this info? Yeah, I might have forgot to add it.
But: IVR Profile, checked; RCS options checked…

So from where does it grabs that URL?

cavagnaro · June 3, 2019, 1:51pm

Ok it grabbed from HTCC itself, externalApiUrlV2…
Now RCS complains:



2019-06-03T08:48:28.487 Dbg 09900 org.eclipse.jetty.server.handler.AbstractHandler starting org.eclipse.jetty.server.Server@58a5be95
2019-06-03T08:48:28.487 Dbg 09900 org.eclipse.jetty.util.component.AbstractLifeCycle starting qtp683021885{8<=0<=0/254,-1}
2019-06-03T08:48:28.487 Dbg 09900 org.eclipse.jetty.util.component.AbstractLifeCycle STARTED qtp683021885{8<=3<=8/254,0}
2019-06-03T08:48:28.487 Dbg 09900 org.eclipse.jetty.util.component.AbstractLifeCycle starting SslSocketConnector@0.0.0.0:7001
2019-06-03T08:48:28.491 Trc 09900 org.eclipse.jetty.util.component.AbstractLifeCycle FAILED SslSocketConnector@0.0.0.0:7001: java.lang.IllegalStateException: SSL doesn't have a valid keystore java.lang.IllegalStateException: SSL doesn't have a valid keystore
java.lang.IllegalStateException: SSL doesn't have a valid keystore
       at org.eclipse.jetty.util.ssl.SslContextFactory.checkKeyStore(SslContextFactory.java:1166) 
       at org.eclipse.jetty.server.ssl.SslSocketConnector.doStart(SslSocketConnector.java:356) 
       at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) 
       at org.eclipse.jetty.server.Server.doStart(Server.java:272) 
       at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) 
       at RcsMain.run(RcsMain.java:73) 
       at RcsMain.main(RcsMain.java:65) 


2019-06-03T08:48:28.491 Trc 09900 org.eclipse.jetty.util.component.AbstractLifeCycle FAILED org.eclipse.jetty.server.Server@58a5be95: java.lang.IllegalStateException: SSL doesn't have a valid keystore java.lang.IllegalStateException: SSL doesn't have a valid keystore
java.lang.IllegalStateException: SSL doesn't have a valid keystore
       at org.eclipse.jetty.util.ssl.SslContextFactory.checkKeyStore(SslContextFactory.java:1166) 
       at org.eclipse.jetty.server.ssl.SslSocketConnector.doStart(SslSocketConnector.java:356) 
       at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) 
       at org.eclipse.jetty.server.Server.doStart(Server.java:272) 
       at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) 
       at RcsMain.run(RcsMain.java:73) 
       at RcsMain.main(RcsMain.java:65)

Kubig · June 3, 2019, 2:01pm

If you do not need media encryption, you can use empty keystore file to enable RCS running properly

cavagnaro · June 3, 2019, 2:35pm

The keystore.bin file?

Kubig · June 3, 2019, 3:05pm

Exactly

cavagnaro · June 3, 2019, 3:07pm

So after RCS is started first time and keystore.bin is created, I just replace by an empty file?
Will try

cavagnaro · June 3, 2019, 3:11pm

Seems that is not the issue:



2019-06-03T10:08:42.868 Trc 09900 com.genesyslab.rcs.model.RcsModel Keystore path is: keystore.bin
2019-06-03T10:08:42.872 Trc 09900 com.genesyslab.rcs.model.RcsModel Exception reading keystore: Exception: class java.io.EOFException / null
2019-06-03T10:08:42.872 Alr 09900 com.genesyslab.rcs.controller.RcsWebMvcConfiguration Fatal error - unable to initialize cryptography, exiting now! java.io.EOFException
java.io.EOFException:

In previous standard keystore.bin that layer was passed normally. Seems another thing to do with SSL…but where?

Kubig · June 3, 2019, 3:22pm

It seems that the keystore file is broken. Try to create a new one or use any existing keystore and point RCS to use it

cavagnaro · June 3, 2019, 3:42pm

Deleted files and let RCS create them again (webapp, jsp and keystore.bin)
Same problem with



2019-06-03T08:48:28.491 Trc 09900 org.eclipse.jetty.util.component.AbstractLifeCycle FAILED SslSocketConnector@0.0.0.0:7001: java.lang.IllegalStateException: SSL doesn't have a valid keystore java.lang.IllegalStateException: SSL doesn't have a valid keystore

If I then replace keystore.bin with just an empty file I get:





2019-06-03T10:08:42.868 Trc 09900 com.genesyslab.rcs.model.RcsModel Keystore path is: keystore.bin
2019-06-03T10:08:42.872 Trc 09900 com.genesyslab.rcs.model.RcsModel Exception reading keystore: Exception: class java.io.EOFException / null
2019-06-03T10:08:42.872 Alr 09900 com.genesyslab.rcs.controller.RcsWebMvcConfiguration Fatal error - unable to initialize cryptography, exiting now! java.io.EOFException
java.io.EOFException:

So or I didn’t understand what to do with the keystore.bin or that just doesn’ work and the issue is another. ???

hsujdik · June 3, 2019, 8:10pm

Try creating an empty keystore:

→ open a command window at the path where the new keystore will be created;

keytool -genkey -alias tempkey -keystore "keystore.bin"

→ set the keystore password while creating it. The default password configured on RCS is “genesys” (without the quotes);
→ delete the temporary certificate after the keystore is created

keytool -delete -alias tempkey -keystore "keystore.bin"

cavagnaro · June 7, 2019, 3:52am

Well issue was actually that the communication property at CME > App Object > Server info > Port > default had to be “http”…which for I have search is not documented…
Thanks guys for all the new information I learned

Kubig · June 7, 2019, 7:52am

FYI : It is documented →
https://docs.genesys.com/Documentation/CR/latest/Solution/RCS#t-6

Configuring an Unsecured (HTTP) Port
Enter the value http in the Connection Protocol field.
Enter the value unsecured in the Listening Mode field.
Leave the other fields empty and click Save.

cavagnaro · June 7, 2019, 1:54pm

I see… Thanks… My bad in assumption that by default should be http basically
Is skipped that point as did not need https.
Thanks

Enviado de meu SM-G9650 usando o Tapatalk