Important tips to be successful and avoid issues in the production environment - 

- Work on a copy of the production environment (with data). Otherwise, time and effort will be spent on issues that the non-production environments have but the production environment does not have. 
- When taking a backup of the environment, please make sure all the following are backed up while the system is stopped. If using Virtual Machines (VMs), we highly recommend taking snapshots - 
  + Filesystem - TIBCO_HOME  - from all servers including ActiveMatrix (AMX) Administrator / EMS. Preserve timestamps, please. 
  + Filesystem - CONFIG_HOME - from all servers including AMX Administrator / EMS. Preserve timestamps, please. 
  + Filesystem - SHARED_MOUNT used by AMX / BPM / EMS (as applicable). Preserve timestamps, please. 
  + Database - AMX (required) / BPM (required) / EMS (as applicable)

The resolution section of this article covers steps that are required to be performed before/after the upgrade process is complete. 

This article summarizes steps that are needed to avoid issues during the upgrade process and/or post-upgrade.

ALL

The following upgrade steps serve as a supplement to the product documentation and should not be treated as a replacement. 

==============  
A. Prior to the Upgrade - (this can be done weeks before the scheduled upgrade - as long as there are no app upgrades/deployments)
==============  

1. Review the documentation - 

   - 4.3.0 readme
   - 4.3.0 system requirements 
   - Release Notes, most importantly Closed Issues for the versions you will be covering in the upgrade. i.e. if upgrading from 3.1 to 4.3,  then go thru 4.0.0, 4.1.0, 4.2.0 & 4.3.0.
   - Upgrade documentation - 
     + Upgrade Introduction & Impact
     + Upgrade Overview & Steps
   - Review the latest hotfix readme of both AMX v3.4.0 and BPM v4.3.0 -- Upgrading to the latest hotfix is recommended. 

2. Housekeeping (should be done regularly) -  All of the below practices will help with the mandatory post-upgrade task of event collector (EC) data migration  
     - List how many older versions of any app exists. Validate with business if any of the older process instances (PI) are still needed. If not, then cancel them. 
     - Remove any older DAAs/templates from Admin > Software Management -- before this check that the version has been removed from the BPM database and node runtime
     - Check if there pending messages on EMS queues. Queue's with pattern *_ErrorQueue do not have any receivers. Messages can get accumulated over a period of time, due to TX failures. If there are no ongoing runtime issues, then purge such queues.
     - Purge EC data - both instance & non-instance. 

3. Make sure the system requirements are still met. System configuration may have changed from what it should be -
   - Unix user limits - open file / max user processes etc
   - Oracle - open cursors

Please refer to the System Requirements listed here - https://docs.tibco.com/pub/amx-bpm/4.3.0/doc/html/install/GUID-7AD2BF6C-A696-40D4-B066-EF5F0033A334.html?scroll=GUID-7AD2BF6C-A696-40D4-B066-EF5F0033A334

4. Perform the following validation to check if any older application versions are not in Prepare For Undeploy (PFU) state - 

https://support.tibco.com/s/article/TIBCO-ActiveMatrix-BPM-product-upgrade-may-fail-while-deploying-BPM-applications

5. Check that all applications are in the correct status (i.e. Running & In Sync). If there are any applications which are not needed, please remove them before performing the upgrade. 

6. Increase the http connection timeout as per - https://support.tibco.com/s/article/TIBCO-Configuration-Tool-TCT-timeout-in-BPM-4-3-upgrade

7. Validate whether Stats Collector (SC) data is used or not. If not, SC should be disabled. Refer - https://support.tibco.com/s/article/How-to-disable-Statistics-Collector

8. Determine if the Content Management part of the BPM data model has been created in the BPM database - 

e.g.:- For Oracle you may use the query - select object_name, object_type from user_objects where object_name like 'CM%';

If it exists, check if it is being used. If inactive, "select * from cm_sequences;" would show values of 1 throughout. 

9. Under SystemEnvironment, check if any of the following applications exist and are being used - 

com.tibco.amx.commonlogging.logservice.app
com.tibco.amx.commonlogging.payloadservice.app
com.tibco.amx.mcr.aggregator

If these applications are not used, then contact TIBCO Support to remove them. 

10. When upgrading from 4.x.x to 4.3.0, the appdev component may fail to start. This issue has been fixed in 4.3.0 HF 1. When upgrading to 4.3.0 HF 1 or higher from a pre-4.3.0 version, this issue will not occur. Please refer the details here - 

https://support.tibco.com/s/article/Appdev-component-fails-to-start-while-upgrading-TIBCO-ActiveMatrix-BPM-4-x-x-to-4-3-0

11. Please remove any engineering build (EB) applied. An upgrade with an AMX or BPM EB applied is not supported. 

12. Ensure only the required features are installed on each node. From AMX Administrator > Infrastructure > Nodes > (select each node) - (bottom-panel) Configuration tab - Features, all the features must be in a "Installed" state. No feature should be in "Marked for install" or "Marked for uninstall". Please repeat this step for each AMX Environment. e.g. - BPMEnvironment & SystemEnvironment. 

13. Ensure, on each server, only a valid tibcohost instance directory exists under CONFIG_HOME/tibcohost/. i.e. CONFIG_HOME/tibcohost/ must contain sub-directories that lead to a tibcohost executable. The folder structure must be - CONFIG_HOME/tibcohost/<tibcoHostInstanceFolder>/host/bin/tibcohost.

If there are any sub-directories like tibamx_patchmgr under CONFIG_HOME/tibcohost/, TCT will detect the CONFIG_HOME to be "NOT ELIGIBLE" for upgrade. Any non-tibcohost directories must be removed from CONFIG_HOME/tibcohost/.

==============  
B. Pre-upgrade - (prior to running TIBCO Configuration Tool [TCT] to upgrade)
==============  

1. Perform all the pre-upgrade steps - https://docs.tibco.com/pub/amx-bpm/4.3.0/doc/html/install/GUID-66B899AB-39A9-452E-B211-1C04DAF8272D.html
2. Install the latest hotfixes for both AMX Platform v3.4.0 and BPM v4.3.0 in the same TIBCO_HOME. 
3. Increase SystemNode & TCT max heap to 4G & MaxPermSize to 2G (if feasible)


==============  
C. Upgrade - 
==============  

1. Perform AMX upgrade using TIBCO_HOME/bpm/4.3/bin/tct > Edit ActiveMatrix BPM > Upgrade or Downgrade TIBCO ActiveMatrix wizard, from each server. 
   - Upgrade the Admin Hosts first
   - Re-launch TCT from each server to upgrade AMX for all remaining hosts. Please note Admin hosts should remain running. 

2. Verify AMX upgrade is successful - 
   - TCT output shows "Build Successful"
   - AMX Administrator is accessible
   - After logging in into AMX Administrator - Infrastructure > Hosts - Verify all hosts are Running and In Sync. The version column lists 3.4.0. If an AMX Platform hotfix was installed in the pre-upgrade stage, the hotfix number will also be listed here. 
   - Similarly, Infrastructure > Nodes (toggle the environment drop-down) - All nodes are Running and In Sync. The version column lists 3.4.0 and if any AMX Platform hotfix is installed.
   - All Applications under SystemEnvironment should be Running and In Sync
  - Under BPM Environment, it is expected to see - 
    - amx.bpm.app in a Stopped or Start Failed state
    - all custom BPM applications are in Start Failed state
 
3. If  AMX Platform 3.4.0 HF 1 or lower is being used, then perform the steps documented here - https://support.tibco.com/s/article/After-Upgrading-To-TIBCO-ActiveMatrix-ServiceGrid-3-4-0-TIBCO-ActiveMatrix-BPM-4-3-0-Old-Versions-Of-User-BPM-Apps-Features-Are-Missing-in-Administrator-Admin-DB

4. Take a full environment backup as explained in the Details section of this article.

5. Perform BPM upgrade using TIBCO_HOME/bpm/4.3/bin/tct > Edit ActiveMatrix BPM > Upgrade TIBCO ActiveMatrix BPM Instance wizard, from the Primary Admin server. 

==============  
D. Post-upgrade - 
==============

1. Validate the following right after the upgrade - 
   a. From AMX Admin - make sure Hosts / Nodes / Applications are In Sync (same as pre-upgrade)
   b. Login into Workspace / Openspace

2. Go through all the post-upgrade tasks - https://docs.tibco.com/pub/amx-bpm/4.3.0/doc/html/install/GUID-901D5F8E-E55D-49CF-AE00-C5DE79037940.html

3. Verify if the below issues exist in the environment -

https://support.tibco.com/s/article/Process-Instance-creation-can-fail-with-a-NullPointerException-after-applying-TIBCO-ActiveMatrix-BPM-4-2-0-HF-2-or-4-1-0-HF-8-or-4-0-0-HF-4
https://support.tibco.com/s/article/No-rows-may-exist-in-PVM-BG-JOBS-after-an-upgrade-to-TIBCO-ActiveMatrix-BPM-4-3-0-hotfix-1

4. During the upgrade, the bindings corresponding to 4.3.0 version are installed with SOAP v1.1. If SOAP v1.2 was being used before the upgrade, then this configuration change needs to be repeated post-upgrade -https://docs.tibco.com/pub/amx-bpm/4.3.0/doc/html/install/GUID-A955485B-5FEE-48FE-918B-EFC1C8D3C439.html

5. Ensure the old SOAP API has been replaced with the new 4.3.0 version. This will apply to any client (e.g. - BW / Mediation / BPM app etc) which invokes amx.bpm.app APIs.