After upgrading TIBCO ActiveMatrix to 3.4.0 GA or 3.4.0 HF01, the old versions of user BPM apps features are missing in administrator (Admin DB), though they are still present in the node runtime (by checking with command tibcohost dipf and depf).

This issue of some versions app features get missing is not expected, and we have filed JIRA AMX-17753 against it. The problem is that during "Upgrade TIBCO ActiveMatrix", TCT would upgrade all versions features into one version for some of BPM user applications. And you will see messages like below in SystemNode log during this procedure.
**********
[Admin Status Events Dispatcher] [INFO ] [] com.tibco.amf.admin.api.amx.lifecycle.impl.NodeStatusNotificationProcessor - Updating version for Feature: com.tibco.userapplication.test from version: 1.0.0.20181009141416108 to version: 1.0.0.20161106072716849
[Admin Status Events Dispatcher] [INFO ] [] com.tibco.amf.admin.api.amx.lifecycle.impl.NodeStatusNotificationProcessor - Updating version for Feature: com.tibco.userapplication.test from version: 1.0.0.20161025152359949 to version: 1.0.0.20161106072716849
**********

Symptom and Impact:
The versions of user BPM apps features get missing in Administrator DB. During next steps of Upgrading TIBCO ActiveMatrix BPM Instance, there is a step of "Reprovisioning all BPM Nodes". This step will try to push feature configuration to runtime node and try to disable those features. However, since those features are still used by applications (old versions feature being used by the outstanding process instances), TCT be hanging forever and lead to the 45 mins timeout exception, and eventually TCT aborts with error below:
**********
18 Oct 2019 17:54:28  WARN - The current action ('reprovision') has timed out since runtime has not completed any tasks for this action in the last 45 minutes. Note that only the CLI action has timed out, and the runtime tasks will continue in the background and can be monitored from Administrator UI.
18 Oct 2019 17:54:28 ERROR - TIBCO-AMX-CLI-000043: Action timed out

BUILD FAILED
/opt/tibco/tibco/data/tct/bpmupgrade/2019-10-18-16-46-27/scripts/build.xml:36: The following error occurred while executing this line:
/opt/tibco/tibco/data/tct/bpmupgrade/2019-10-18-16-46-27/scripts/build.xml:14: The following error occurred while executing this line:
/opt/tibco/bpm/4.3/scripts/upgrade/build.xml:584: The following error occurred while executing this line:
/opt/tibco/bpm/4.3/scripts/instance-mgmt.xml:830: The following error occurred while executing this line:
/opt/tibco/bpm/4.3/scripts/instance-mgmt.xml:686: The following error occurred while executing this line:
/opt/tibco/bpm/4.3/scripts/instance-mgmt.xml:663: The following error occurred while executing this line:
/opt/tibco/bpm/4.3/scripts/deployer/upgrades/upgrade.xml:172: The following error occurred while executing this line:
/opt/tibco/bpm/4.3/scripts/deployer/amx-admin-macros.xml:332: TIBCO-AMX-CLI-000043: Action timed out

Total time: 61 minutes 0 seconds
**********

Verification :
Let us assume that a BPM user application has more than one template version deployed and also the application versions have outstanding PI's before the AMX 3.4.0 upgrade. Once the environment is upgraded to AMX 3.4.0 GA or 3.4.0 HF1, the application feature will show only one version instead of multiple, and doesn't contain any other version features.

We can verify this from AMX Admin GUI -> BPMNode -> Configuration -> Features tab

By running the below commands would give you the features present on the runtime.

./tibcohost dipf
./tibcohost depf -nodeName <NodeName>

After Upgrading To TIBCO ActiveMatrix ServiceGrid 3.4.0/ TIBCO ActiveMatrix BPM 4.3.0, Old Versions Of User BPM Apps Features Are Missing in Administrator (Admin DB)

All Supported Environments

We have a new functionality/feature “Synchronizing Node Feature List” introduced in AMX 3.4.0 GA:

AMSG 3.4.0 Doc: Administration -> Software Management -> Synchromizing Node Feature List

https://docs.tibco.com/pub/amsg/3.4.0/doc/html/GUID-67B9681C-4F7D-43CC-ABD5-3EC642F15D61.html

Please refer point 3) to fix the issue: 
1. Please set the correct Environment/Host/Node names in the node_data.xml file which is located in the samples directory ( CONFIG_HOME\admin\<Enterprise_Name>\samples or TIBCO_HOME\administrator\3.4\samples).
2. Enable the option "syncAllFeatures" for the build target "syncNodeFeaturesStatus" (by default this option is disabled) in the node_build.xml:

<target name="syncNodeFeaturesStatus">
<AMXAdminTask
action="syncNodeFeaturesStatus"
objectSelector="Environment/Node"
remote="true"
propsFile="${instanceProperties}"
dataFile="${dataFile}"
failOnError="false"
options="syncAllFeatures" />
</target>

3. Run the syncNodeFeaturesStatus target via CLI:

ant -f node_build.xml syncNodeFeaturesStatus

Note: This issue has been resolved in AMX 3.4.0 HF02. So this article is applicable only to AMX 3.4.0 GA and HF01.

AMX-17753