How to resolve TML Installer 5.5.1 cassandra Readiness probe failure (nodetool: Malformed IPv6 address) for TIBCO Cloud API Management - Local Edition

How to resolve TML Installer 5.5.1 cassandra Readiness probe failure (nodetool: Malformed IPv6 address) for TIBCO Cloud API Management - Local Edition

book

Article ID: KB0072001

calendar_today

Updated On:

Products Versions
TIBCO Cloud API Management - Local Edition ML 5.5.1

Description

TML Installer 5.5.1 cassandra was not coming up and getting error : Readiness probe failure. Checking if cassandra is ready.
nodetool status throwing exception : 
nodetool: Failed to connect to '127.0.0.1:7199' - URISyntaxException: 'Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199').

/usr/local/bin/nosql-readyprobe.sh file is trying to check the readiness through a nodetool status which is in return throwing a failure for cassandra check, although cassandra is running fine at the background.

This is happening due to java version compatibility issue.

Issue/Introduction

How to resolve TML Installer 5.5.1 cassandra Readiness probe failure (nodetool: Malformed IPv6 address) for TIBCO Cloud API Management - Local Edition

Resolution

Basically, we have come up 2 workarounds for the issue, few customer's issue resolved with the first, but with few both the work arounds were needed to get the cluster up and running.

workaround 1:
Modify install-core-tools.sh in the tml-base image, add the command to install java 1.8.0 before installing Casssandra, to avoid the java compatibility issue as a work around.

Make sure you have access for modification of script inside the container :
docker ps
which shows TML container ID
Get inside the container :
docker exec -ti <conatinerID>/bin/bash

get inside the build files :
cd /var/jenkins_home/docker-build/tmgc/install

ls -ltr lists out the files.
we need to modify install-core-tools.sh

Make a copy of the script file first then modify.
cp install-core-tools.sh install-core-tools.sh_bkp

Open the script in vi and edit as below.
vi install-core-tools.sh

#Add below line 

yum_install java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64
on top of 
yum_install java-11-openjdk java-11-openjdk-devel

:wq!

It clears cassandra readiness check.

work around 2: 
By modifying nosql-ready-probe.sh file at the readiness probe level by adding '-Dcom.sun.jndi.rmiURLParsing=legacy', for nodetool to use the legacy way of connecting to rmi instead of the stricter way introduced in latest java version.

$(nodetool status -Dcom.sun.jndi.rmiURLParsing=legacy | grep $POD_IP)
instead of $(nodetool status | grep $POD_IP)


Build images again and try deploying the cluster, after making sure the new image version is updated in registry(nosql-pod-0.yaml).
Make sure old cluster is deleted.

./create-tml-cluster.sh

Due to constant repo changes regarding java versions (or even other components) it will be hard for us/Engineering to validate every change to the image build process and its impact on deployment.

Note : Customers to do thorough testing on their non-prod clusters with these workarounds before implementing the change in production. 

Powered by Wolken Software