"Service communication error" message in the Automation job activity is seen while executing a job on TIBCO Spotfire Server

"Service communication error" message in the Automation job activity is seen while executing a job on TIBCO Spotfire Server

book

Article ID: KB0075318

calendar_today

Updated On:

Products Versions
Spotfire Automation Services 10.0 and higher

Description

A "Service communication error" is seen in the activity message when the job is not executed due to the server not being able to communicate with TIBCO Spotfire Automation service due to (temporary) connectivity errors. Below is the screenshot for your reference:
User-added image

You may see the following error in server.log:

ERROR 2020-03-31T11:00:44,067-0400 [automation-services-job-933cf742-8405-4bc3-92b3-9fb6fc0c6b62] scheduler.service.ASJobSender: An error occurred while executing job task a07b1a11-bce6-43cb-97bd-02279d3e9d24
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://123.234.345.456:9502/1188f7a8-50fc-4948-bd5f-60c90c266101/AutomationService/StartJob": Connect to 123.234.345.456:9502 [/123.234.345.456] failed: connect timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to 123.234.345.456:9502 [/123.234.345.456] failed: connect timed out
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:751) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:677) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:421) ~[spring-web.jar:5.2.3.RELEASE]
    at com.spotfire.server.scheduler.service.ASJobSender.sendJob(ASJobSender.java:100) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.ScheduledJobsQueueProcessorManager.processQueuedTask(ScheduledJobsQueueProcessorManager.java:95) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.ScheduledJobsQueueProcessorManager.processQueuedTask(ScheduledJobsQueueProcessorManager.java:45) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.AbstractQueueManager.lambda$processQueuedItems$22(AbstractQueueManager.java:719) ~[spotfire-common-services.jar:?]
    at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_241]
    at com.spotfire.server.scheduler.service.AbstractQueueManager.processQueuedItems(AbstractQueueManager.java:717) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.AbstractQueueManager.processQueuedItemsWithinLock(AbstractQueueManager.java:338) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.AbstractQueueManager.internalProcessQueuedItems(AbstractQueueManager.java:311) ~[spotfire-common-services.jar:?]
    at com.spotfire.server.scheduler.service.AbstractQueueManager.lambda$processQueuedItems$2(AbstractQueueManager.java:288) ~[spotfire-common-services.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_241]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_241]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_241]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_241]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_241]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_241]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 123.234.345.456:9502 [/10.220.181.61] failed: connect timed out
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient.jar:4.5.10]
    at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:742) ~[spring-web.jar:5.2.3.RELEASE]
    ... 18 more
Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[?:1.8.0_241]
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[?:1.8.0_241]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_241]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_241]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_241]
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[?:1.8.0_241]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_241]
    at java.net.Socket.connect(Socket.java:606) ~[?:1.8.0_241]
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient.jar:4.5.10]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient.jar:4.5.10]
    at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web.jar:5.2.3.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:742) ~[spring-web.jar:5.2.3.RELEASE]
    ... 18 more

Resolution

Verify if the TIBCO Spotfire Server and Node managers machines can communicate:
  • Check if the TIBCO Spotfire Server can ping and Telnet to the Node Manager server on the ports 9501 or 9502. You can refer to the log snippet and verify the connectivity based on what port it is failing. For example, based on the above error message you can run the syntax "telnet 10.220.181.61 9502" from Spotfire Server machine and verify if you are able to connect. See below screenshot for your reference:User-added image

    If the connection succeeds, a blank screen will show up as below, meaning that the computer port is open:
    User-added image

If it fails with "Connect failed" message, please verify if the port is open and that you are able to communicate:

  •  Check if there is any firewall rules that is blocking the communication between Spotfire Server and Node Manager. 

Issue/Introduction

The "Service communication error" occurs due to the server not being able to access the node at the time due to connectivity issues or network glitch

Additional Information

Doc: TIBCO Spotfire Automation Services Job Scheduling