When installing BPM product or hotfix into a TIBCO_HOME where many other products, previous versions & hotfixes already exist, TUI can fail to complete post-install tasks because of an OutOfMemoryError.

The OOM occurs when a BPM post-install ant script tries to build a classpath with all jars under TIBCO_HOME/components folder; ant's JVM heap is exhausted during the construction of this string.

The post-install failure can be avoided using ANT_OPTS environment variable, but this leads to similar OOM in TUI itself (when it tries to handle the logfile from the post-install script)

To prevent these OutOfMemoryErrors, it is necessary to override the default maximum heap size for JVMs of both ant and TUI.

Note that both steps below are required to avoid the OOMs in this scenario

1. To set an increased maximum heap size for JVM used to run ant tasks:

To set the effective value only for the current shell session:

Linux:  export ANT_OPTS="-Xmx1g $ANT_OPTS"
Windows: set ANT_OPTS="-Xmx1g %ANT_OPTS%"

To set the options permanently:

Linux: add the equivalent into your shell profile script, e.g. echo "export ANT_OPTS=\"-Xmx1g \$ANT_OPTS\"" >> ~/.bashrc
Windows: In Explorer, go to: Control Panel\All Control Panel Items\System
Click Advanced System Settings
Go to Advanced tab
Click Environment Variables
If ANT_OPTS already exists, click Edit... and add -Xmx1g (or replace existing -Xmx value, if present
Otherwise, click New... Variable Name: ANT_OPTS   Variable Value: -Xmx1g

2. Create a file with JVM options alongside the TUI executable

Note: this file must have the same name as the TUI executable, but with extension .ja (instead of .exe / .bin)
Windows: 
cd \path\to\unzippedInstallerFolder
echo -Xmx1g>TIBCOUniversalInstaller-x86-64.ja
.\TIBCOUniversalInstaller-x86-64.exe  followed by any other desired options

Linux:  in a shell, navigate to the folder where the product/hotfix was unzipped
cd /path/to/unzippedInstallerFolder
echo -Xmx1g>TIBCOUniversalInstaller-x86-64.ja
./TIBCOUniversalInstaller-x86-64.bin  followed by any other desired options