Steps to reproduce:

1. Create TIBCO_Home with e.g. BPM 4.3.1 and AMX 3.4.0 HF03. // Note: The issue has NOT been reproduced in pure AMSG 3.4.0 HF3 env.
2. Run TCT to configure the BPM 4.3.1 and AMX 3.4.0 HF03 - create an AMX admin server with Oracle 12c as backend Admin DB. Now the AMX side JDBC driver version is 12.
3. Stop tibcohost and Nodes.
4. Take the VM snapshot here.
5. Run the TCT to configure third party DB driver (new version JDBC Driver - Oracle 19.3c driver).
6. Start the tibcohost.
7. Execute the 'updateJDBCDriver' CLI script to upgrade the JDBC driver from version 12 to 19c, following the below KB article:

https://supportapps.tibco.com/ka/article/744/000040744.htm

8. Expected: SystemNode stopped and restarted up successfully. New version DB driver feature installed and enabled, and old DB driver feature disabled and uninstalled from SystemNode. And the same for remote runtime Nodes.

Observed: SystemNode stopping hang there. The new DB driver (19.3c) feature is in "Marked for Install", and old driver (12) feature is in "Marked for Uninstall" state. See sample SystemNode logs below.

===================
SystemNode logs:

03 Aug 2021 03:06:19,043 [System Bundle Shutdown] [INFO ] [] com.tibco.amx.admin.server - TIBCO-AMX-ADMIN-004058: TIBCO ActiveMatrix Administrator Server is stopping.... 
03 Aug 2021 03:06:19,090 [System Bundle Shutdown] [WARN ] [] org.springframework.beans.factory.support.DisposableBeanAdapter - Couldn't find a destroy method named 'destroy' on bean with name 'nodeCleanupRUTask'
03 Aug 2021 03:06:23,325 [TIBCO EMS Session Dispatcher (38)] [ERROR] [] com.tibco.amf.admin.api.core.task.TaskService - TIBCO-AMX-ADMIN-002063: Exception while executing task.execute
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.tibco.amf.admin.api.core.task.impl.TaskServiceImpl$TaskExecutionWrapper.execute(TaskServiceImpl.java:123)
    at sun.reflect.GeneratedMethodAccessor157.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.tibco.amf.admin.api.amx.deployment.server.impl.JmsDeploymentServer.onTaskToProcess(JmsDeploymentServer.java:716)
    at com.tibco.amf.admin.api.amx.deployment.server.impl.JmsDeploymentServer$JmsTaskQueue$1.onMessage(JmsDeploymentServer.java:1444)
    at com.tibco.tibjms.TibjmsxSessionImp._submit(TibjmsxSessionImp.java:4366)
    at com.tibco.tibjms.TibjmsxSessionImp._dispatchAsyncMessage(TibjmsxSessionImp.java:2298)
    at com.tibco.tibjms.TibjmsxSessionImp$Dispatcher.run(TibjmsxSessionImp.java:3763)
Caused by: com.tibco.amf.admin.api.amx.deployment.server.QueueOfflineException
    at com.tibco.amf.admin.api.amx.node.impl.NodeEnableRUTask.execute(NodeEnableRUTask.java:106)
    ... 13 more
03 Aug 2021 03:06:23,325 [TIBCO EMS Session Dispatcher (38)] [WARN ] [] com.tibco.amf.admin.api.amx.deployment.server.impl.JmsDeploymentServer - The task could not be processed, the host seems to be offline
 
03 Aug 2021 03:07:41,606 [TIBCO EMS Session Dispatcher (162)] [INFO ] [] com.tibco.amx.admin.api.node - TIBCO-AMX-ADMIN-002136: Successfully executed task identified by task ID '10,924' and user action [root:Node-Change features:10922]: Provision release units to node 'BPMNode' of host 'BPMHost'
03 Aug 2021 03:07:41,622 [TIBCO EMS Session Dispatcher (162)] [INFO ] [] com.tibco.amf.admin.api.core.task.TaskService - TIBCO-AMX-ADMIN-002005: Adding task task_id:'10926' to QueueableTaskThreadLocal as it has no more dependencies.
 
03 Aug 2021 04:05:23,099 [Thread-1250] [ERROR] [] com.tibco.amx.admin.api.node - TIBCO-AMX-ADMIN-024128: Error from runtime when REPROVISION node SystemNode to host SystemHost
TIBCO-AMX-HPA-API-000212: error from remote host
    at com.tibco.amf.hpa.core.client.api.impl.JMXConnectionState.invokeMBean(JMXConnectionState.java:243)
    at com.tibco.amf.hpa.core.client.api.impl.HostManagerImpl.invokeMBean(HostManagerImpl.java:1795)
    at com.tibco.amf.hpa.core.client.api.impl.HostManagerImpl.provisionProductFeatures(HostManagerImpl.java:1519)
    at com.tibco.amf.admin.api.amx.node.impl.NodeEnableRUTask.doReprovisionToRuntime(NodeEnableRUTask.java:404)
    at com.tibco.amf.admin.api.amx.node.impl.NodeEnableRUTask.executeTask(NodeEnableRUTask.java:135)
    at com.tibco.amf.admin.api.amx.node.impl.NodeEnableRUTask.access$2(NodeEnableRUTask.java:126)
    at com.tibco.amf.admin.api.amx.node.impl.NodeEnableRUTask$CustomThread.run(NodeEnableRUTask.java:466)
Caused by: java.io.InterruptedIOException: Waiting response timeout: 3600000
    at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.sendWithReturn(ClientSynchroMessageConnectionImpl.java:263)
    at javax.management.remote.generic.ClientIntermediary.mBeanServerRequest(ClientIntermediary.java:975)
    at javax.management.remote.generic.ClientIntermediary.mBeanServerRequest(ClientIntermediary.java:957)
    at javax.management.remote.generic.ClientIntermediary.invoke(ClientIntermediary.java:478)
    at javax.management.remote.generic.GenericConnector$RemoteMBeanServerConnection.invoke(GenericConnector.java:522)
    at com.tibco.amf.hpa.core.client.api.impl.JMXConnectionState.invokeMBean(JMXConnectionState.java:231)
    ... 6 more
03 Aug 2021 04:05:23,209 [Thread-1250] [ERROR] [] com.tibco.amx.admin.api.node - TIBCO-AMX-ADMIN-024129: Error when enabling/disabling features node SystemNode to host SystemHost
com.tibco.amf.admin.api.core.exception.AdminException
summary = Please see the cause for more details
 
    at com.tibco.amf.admin.api.amx.node.impl.NodeEnableRUTask.executeTask(NodeEnableRUTask.java:154)
    at com.tibco.amf.admin.api.amx.node.impl.NodeEnableRUTask.access$2(NodeEnableRUTask.java:126)
    at com.tibco.amf.admin.api.amx.node.impl.NodeEnableRUTask$CustomThread.run(NodeEnableRUTask.java:466)
Caused by: TIBCO-AMX-HPA-API-000212: error from remote host
    at com.tibco.amf.hpa.core.client.api.impl.JMXConnectionState.invokeMBean(JMXConnectionState.java:243)
    at com.tibco.amf.hpa.core.client.api.impl.HostManagerImpl.invokeMBean(HostManagerImpl.java:1795)
    at com.tibco.amf.hpa.core.client.api.impl.HostManagerImpl.provisionProductFeatures(HostManagerImpl.java:1519)
    at com.tibco.amf.admin.api.amx.node.impl.NodeEnableRUTask.doReprovisionToRuntime(NodeEnableRUTask.java:404)
    at com.tibco.amf.admin.api.amx.node.impl.NodeEnableRUTask.executeTask(NodeEnableRUTask.java:135)
    ... 2 more
Caused by: java.io.InterruptedIOException: Waiting response timeout: 3600000
    at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.sendWithReturn(ClientSynchroMessageConnectionImpl.java:263)
    at javax.management.remote.generic.ClientIntermediary.mBeanServerRequest(ClientIntermediary.java:975)
    at javax.management.remote.generic.ClientIntermediary.mBeanServerRequest(ClientIntermediary.java:957)
    at javax.management.remote.generic.ClientIntermediary.invoke(ClientIntermediary.java:478)
    at javax.management.remote.generic.GenericConnector$RemoteMBeanServerConnection.invoke(GenericConnector.java:522)
    at com.tibco.amf.hpa.core.client.api.impl.JMXConnectionState.invokeMBean(JMXConnectionState.java:231)
    ... 6 more
===================

Root cause:

Due to an AMX side JMS client library issue the SystemNode JVM process cannot be shut down successfully when executing the 'updateJDBCDriver' CLI script. Consequently, neither SystemNode nor the runtime Nodes get their JDBC driver version upgraded as expected.

All Supported OS Platform. Oracle client driver upgrade from version 12 to 19c

Workaround #1: 

1. Once running into this issue AMX-18316 upon executing CLI command 'updateJDBCDriver', kill the SystemNode OS process
2. Start SystemNode (and restart SystemNodeReplica) with -clearCache option
3. Click to "Apply" the JDBC driver feature of each BPM Node from Infrastructure --> Node --> Configuration --> Features

Workaround #2:

1. Stop the TIBCO Host instance.

2. In TIBCO Configuration Tool, run the 'Configure Third-Party Driver' Wizard to configure the TIBCO-enabled JDBC driver for your Oracle version JDBC driver with the compatible driver jar file.

3. Start the TIBCO Host instance.

4. Assuming that all SystemHost/SystemNode, SystemHostReplica/SystemNodeReplica (if applicable), and remote hosts/Nodes are up and running.

5. Manually upgrade jdbc driver separately for SystemNode. This can be done as below:

In ActiveMatrix Administrator:
a) Select Infrastructure > Environment and select SystemEnvironment.
b) Go to Admin GUI -> Infrastructure -> Node -> Configuration -> Features section, find the TIBCO enabled JDBC driver for Oracle entry.
c) In the Version dropdown Select the new version i.e. disable the old jdbc driver, and enable the new jdbc driver.
d) Click Save.
e) Click "Apply with Resolve" for the features change to take effect.

6. Make sure that the new jdbc driver is successfully enabled, and the old jdbc driver disabled from SystemNode.

7. Repeat these steps from 1 to 5 for SystemNodeReplica if applicable.

8. Execute the 'updateJDBCDriver' CLI script to upgrade the JDBC driver for all other remote hosts and Nodes. This CLI execution will be a no-op for SystemNode whose driver has already been upgraded manually.


Compared with Workaround #1 where we need to manually click buttons to "Apply" the new JDBC driver feature to each of the runtime Nodes, Workaround #2 step #8 is just a single CLI execution. So for large size environments the approach #2 is less labor-intensive and more efficient and user friendly.

For a fully automated approach of upgrading JDBC driver (i.e., running 'updateJDBCDriver' will take care of the upgrade of both SystemNode/SystemNodeReplica and all other runtime Nodes), we may need JMS client jar upgrade which will be considered in future AMX 3.4.x or 3.x releases.

SystemNode restart hungs upon 'updateJDBCDriver' CLI execution

AMX-18316