"Service communication error" message in the Automation job activity is seen while executing a job on TIBCO Spotfire Server
book
Article ID: KB0075318
calendar_today
Updated On:
Products
Versions
Spotfire Automation Services
10.0 and higher
Description
A "Service communication error" is seen in the activity message when the job is not executed due to the server not being able to communicate with TIBCO Spotfire Automation service due to (temporary) connectivity errors. Below is the screenshot for your reference:
You may see the following error in server.log:
ERROR 2020-03-31T11:00:44,067-0400 [automation-services-job-933cf742-8405-4bc3-92b3-9fb6fc0c6b62] scheduler.service.ASJobSender: An error occurred while executing job task a07b1a11-bce6-43cb-97bd-02279d3e9d24 org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://123.234.345.456:9502/1188f7a8-50fc-4948-bd5f-60c90c266101/AutomationService/StartJob": Connect to 123.234.345.456:9502 [/123.234.345.456] failed: connect timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to 123.234.345.456:9502 [/123.234.345.456] failed: connect timed out at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:751) ~[spring-web.jar:5.2.3.RELEASE] at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:677) ~[spring-web.jar:5.2.3.RELEASE] at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:421) ~[spring-web.jar:5.2.3.RELEASE] at com.spotfire.server.scheduler.service.ASJobSender.sendJob(ASJobSender.java:100) ~[spotfire-common-services.jar:?] at com.spotfire.server.scheduler.service.ScheduledJobsQueueProcessorManager.processQueuedTask(ScheduledJobsQueueProcessorManager.java:95) ~[spotfire-common-services.jar:?] at com.spotfire.server.scheduler.service.ScheduledJobsQueueProcessorManager.processQueuedTask(ScheduledJobsQueueProcessorManager.java:45) ~[spotfire-common-services.jar:?] at com.spotfire.server.scheduler.service.AbstractQueueManager.lambda$processQueuedItems$22(AbstractQueueManager.java:719) ~[spotfire-common-services.jar:?] at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_241] at com.spotfire.server.scheduler.service.AbstractQueueManager.processQueuedItems(AbstractQueueManager.java:717) ~[spotfire-common-services.jar:?] at com.spotfire.server.scheduler.service.AbstractQueueManager.processQueuedItemsWithinLock(AbstractQueueManager.java:338) ~[spotfire-common-services.jar:?] at com.spotfire.server.scheduler.service.AbstractQueueManager.internalProcessQueuedItems(AbstractQueueManager.java:311) ~[spotfire-common-services.jar:?] at com.spotfire.server.scheduler.service.AbstractQueueManager.lambda$processQueuedItems$2(AbstractQueueManager.java:288) ~[spotfire-common-services.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_241] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_241] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_241] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_241] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_241] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_241] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241] Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 123.234.345.456:9502 [/10.220.181.61] failed: connect timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) ~[httpclient.jar:4.5.10] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[httpclient.jar:4.5.10] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient.jar:4.5.10] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient.jar:4.5.10] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient.jar:4.5.10] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient.jar:4.5.10] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient.jar:4.5.10] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient.jar:4.5.10] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient.jar:4.5.10] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient.jar:4.5.10] at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web.jar:5.2.3.RELEASE] at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web.jar:5.2.3.RELEASE] at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web.jar:5.2.3.RELEASE] at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:742) ~[spring-web.jar:5.2.3.RELEASE] ... 18 more Caused by: java.net.SocketTimeoutException: connect timed out at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[?:1.8.0_241] at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[?:1.8.0_241] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_241] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_241] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_241] at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[?:1.8.0_241] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_241] at java.net.Socket.connect(Socket.java:606) ~[?:1.8.0_241] at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368) ~[httpclient.jar:4.5.10] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient.jar:4.5.10] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[httpclient.jar:4.5.10] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient.jar:4.5.10] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient.jar:4.5.10] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient.jar:4.5.10] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient.jar:4.5.10] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient.jar:4.5.10] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient.jar:4.5.10] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient.jar:4.5.10] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient.jar:4.5.10] at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web.jar:5.2.3.RELEASE] at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web.jar:5.2.3.RELEASE] at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web.jar:5.2.3.RELEASE] at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:742) ~[spring-web.jar:5.2.3.RELEASE] ... 18 more
Resolution
Verify if the TIBCO Spotfire Server and Node managers machines can communicate:
Check if the TIBCO Spotfire Server can ping and Telnet to the Node Manager server on the ports 9501 or 9502. You can refer to the log snippet and verify the connectivity based on what port it is failing. For example, based on the above error message you can run the syntax "telnet 10.220.181.61 9502" from Spotfire Server machine and verify if you are able to connect. See below screenshot for your reference:
If the connection succeeds, a blank screen will show up as below, meaning that the computer port is open:
If it fails with "Connect failed" message, please verify if the port is open and that you are able to communicate:
Check if there is any firewall rules that is blocking the communication between Spotfire Server and Node Manager.
Issue/Introduction
The "Service communication error" occurs due to the server not being able to access the node at the time due to connectivity issues or network glitch