Resolution:
While running BW_application on AIX environment below error is thrown:
**************************
[java.lang.OutOfMemoryError]: [Failed to fork OS thread at java.lang.Thread.startImpl (Native Method)]
        at com.tibco.plugin.java.JavaActivity.eval(JavaActivity.java:395)
        at com.tibco.pe.plugin.Activity.eval(Activity.java:209)
        at com.tibco.pe.core.TaskImpl.eval(TaskImpl.java:540)
        at com.tibco.pe.core.Job.a(Job.java:712)
        at com.tibco.pe.core.Job.k(Job.java:502)
        at com.tibco.pe.core.JobDispatcher$JobCourier.a(JobDispatcher.java:249)
        at com.tibco.pe.core.JobDispatcher$JobCourier.run(JobDispatcher.java:200)

**************************

Setting the below property on AIX environment helps to resolve the issue:

********
tibco.env.LDR_CNTRL=MAXDATA=0x40000000
********

Set the above property in the tra file; set MAXDATA to lower value, if you need to set the heap to a larger value.

Please also refer to the below links as it explains how to tune java apps to allocate more memory on AIX platform and ‘LDR_CNTRL’ environment variable settings information resp.

1. http://www-128.ibm.com/developerworks/eserver/articles/aix4java1.html

2. http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.itame3.doc_5.1/am51_perftune113.htm

Likewise setting java max heap size beyond 512 mb on AIX requires setting above mentioned property to be set in the tra.

This property can be added into the bwengine.xml located at &ltinstall_dir>\&ltTIBCO_home>\bw\&ltbw_version>\lib\com\tibco\deployment\
As shown below before creating the ear

To change the default values for any deployment specify a new value in the Value column.
**********
&ltproperty>
       &ltname&gtAIX tuning property for heap size allocation</name>
       &ltoption&gttibco.env.LDR_CNTRL=MAXDATA</option>
       &ltdefault&gt0x40000000</default>
       &ltdescription&gtset MAXDATA to lower value for higher heap value</description>
</property>
**********

How to enable more Java memory for BW on IBM AIX machines to avoid 'OOM' errors?