TERR service installed on Linux may fail to start due to old Docker version being used.

TERR service installed on Linux may fail to start due to old Docker version being used.

book

Article ID: KB0073536

calendar_today

Updated On:

Products Versions
Spotfire Enterprise Runtime for R - Server Edition All Versions

Description

When installing a new TERR service on a Linux-based Node Manager (see Installing the TERR service instance on a node manager for a Spotfire Server for more details), the TERR Service may fail to start with a generic error seen in the TERR service logs captured in DEBUG mode:
DEBUG 2019-08-31T19:56:26,230 [ImageBuilder] topology.utility.DockerImageBuilder: imageName is terrsrv:1433763735n
DEBUG 2019-08-31T19:56:27,321 [terrsrv-61003-dockerEngine-start-sequence] topology.utility.DockerImageBuilder: imageName is terrsrv:1433763735n
DEBUG 2019-08-31T19:56:27,522 [terrsrv-61003-create-docker-container] topology.engine.EngineControllerInitializer: Container c01f1f70dd82c90287c2041894f914721ece8b0911bdcd4906e1e099696bb502 created.
DEBUG 2019-08-31T19:56:28,296 [terrsrv-61003-c01f1-start-docker-container] topology.engine.EngineControllerInitializer: Container c01f1f70dd82c90287c2041894f914721ece8b0911bdcd4906e1e099696bb502 started for terrsrv-61003-c01f1.
WARN  2019-08-31T19:56:29,166 [terrsrv-61003-c01f1-init-startExecutionEngine] topology.engine.EngineControllerInitializer: Unhandled exception java.io.IOException: Connection reset by peer was caught while engine was INIT_START_EXECUTION_ENGINE.
ERROR 2019-08-31T19:56:29,166 [terrsrv-61003-c01f1-init-startExecutionEngine] topology.engine.EngineControllerInitializer: Failed to start terr-service-remote.jar in terrsrv-61003-c01f1
java.lang.RuntimeException: java.io.IOException: Connection reset by peer
    at com.google.common.base.Throwables.propagate(Throwables.java:241) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.spotify.docker.client.DefaultLogStream.computeNext(DefaultLogStream.java:58) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.spotify.docker.client.DefaultLogStream.computeNext(DefaultLogStream.java:35) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.spotify.docker.client.DefaultLogStream.readFully(DefaultLogStream.java:77) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.tibco.spotfire.topology.engine.EngineControllerInitializerDocker.dockerExecStart(EngineControllerInitializerDocker.java:366) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.tibco.spotfire.topology.engine.EngineControllerInitializerDocker.execCreateAndStart(EngineControllerInitializerDocker.java:351) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.tibco.spotfire.topology.engine.EngineControllerInitializerDocker.startRemoteEngine(EngineControllerInitializerDocker.java:79) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.tibco.spotfire.topology.engine.EngineControllerInitializer.lambda$startRemoteEngineOnNewThread$0(EngineControllerInitializer.java:120) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_212]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: java.io.IOException: Connection reset by peer
    at jnr.enxio.channels.Common.read(Common.java:67) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at jnr.enxio.channels.AbstractNativeSocketChannel.read(AbstractNativeSocketChannel.java:62) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at jnr.unixsocket.UnixSocketChannel.read(UnixSocketChannel.java:250) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[?:1.8.0_212]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:1.8.0_212]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:1.8.0_212]
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:197) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at org.apache.http.impl.io.IdentityInputStream.read(IdentityInputStream.java:85) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_212]
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_212]
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_212]
    at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:1.8.0_212]
    at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:1.8.0_212]
    at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:102) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream.read(ReaderInterceptorExecutor.java:296) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.google.common.io.ByteStreams.read(ByteStreams.java:898) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.spotify.docker.client.LogReader.nextMessage(LogReader.java:49) ~[terr-service-remote-1.2.0-38.jar!/:?]
    at com.spotify.docker.client.DefaultLogStream.computeNext(DefaultLogStream.java:56) ~[terr-service-remote-1.2.0-38.jar!/:?]
    ... 15 more

WARN  2019-08-31T19:56:29,175 [terrsrv-61003-c01f1-init-startExecutionEngine] topology.engine.AbstractEngineController: Marking terrsrv-61003-c01f1 in state INIT_START_EXECUTION_ENGINE as a failure: Failed to start terr-service-remote.jar
INFO  2019-08-31T19:56:31,194 [EngineQueueHandler-1] topology.engine.EngineCreator: Destroying engine terrsrv-61003-c01f1 that was not owned and in state FAILED
DEBUG 2019-08-31T19:56:31,195 [EngineQueueHandler] topology.engine.EngineQueueHandler: Shutting down terrsrv-61003-c01f1
The error mentioned above and seen in the TERR service logs is a generic error and does not give any specific clues to the root cause and may be seen due to several other reasons. One main reason that this can be seen is the use of an unsupported Docker version on the Linux machine.
 

Issue/Introduction

This article describes steps that need to be followed if TERR service on Linux does not start due to unsupported Docker version installed on the machine.

Environment

OS: Linux

Resolution

When the TERR service is installed on a Linux computer it is configured to use a Docker container. It is a prerequisite to have a supported Docker version installed on the Linux machine before installing a TERR service on the Node Manager. To know the supported Docker versions for a given TERR service version, see the System Requirements page for given TERR service version. For example, for TERR service version 1.2.1 the supported Docker versions are 17.03.1-ce, 17.06.2-ce, 17.09.1-ce, or 17.12.1-ce (17.12.1-ce is recommended)

If the Docker version is not a supported version then installing TERR service from Nodes & Services may fail with this error and the TERR service under Nodes & Services will appear in Red.

To verify the current Docker version, run the following command:
docker version
If the Docker version returned by the above command is not a supported version, then update the version by performing the following steps:

1. Shut down the TERR Service node.
2. Delete the instance of TERR Service.
3. Delete the old Docker images:  
docker rmi -f $(docker images terrsrv -q)

4. Delete the version of Docker that is currently installed and upgrade to a specific newer version of Docker. Involve your Linux administrators if required:
# remove all old versions of docker and add repos and correct version of docker

sudo yum remove docker \
                 docker-client \
                 docker-client-latest \
                 docker-common \
                 docker-latest \
                 docker-latest-logrotate \
                 docker-logrotate \
                 docker-engine

sudo rm -rf /var/lib/docker
sudo yum install -y yum-utils \
 device-mapper-persistent-data \
 lvm2
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

sudo yum install docker-ce-17.12.1.ce –y

# Above command is to install recommended Docker version 17.12.1-ce for TERR service 1.2.1. Version on this command need to be changed as per System requirement page for given TERR service version.

sudo systemctl start docker

# make sure docker starts on every system startup.
sudo systemctl enable docker

# verify that docker is now correct version
sudo docker version

5. Install new TERR instance from Nodes & Services.

Note: Docker is available under separate software license terms and is not part of the TIBCO Spotfire Server or the TERR service. As such, Docker is not within the scope of your license for Spotfire Server or the TERR service. Docker is not supported, maintained, or warranted in any way by TIBCO Software Inc. Download and use of Docker is solely at your own discretion and subject to license terms applicable to Docker.

Additional Information

Doc: Installing the TERR service instance on a node manager for a Spotfire Server Doc: System requirement page for TERR service 1.2.1 Doc: Containerized TERR service :