Resolution:
Tibco BusinessWorks are primary Java applications, Hotspot VM can significantly affect the performance.
You can refer to the following documentation from Sun for Garbage Collection Tuning:
http://java.sun.com/javase/technologies/hotspot/gc/index.jsp

There are some very good tuning recommendations in Sun's Memory Management Whitepaper (for J2SE 5.0 and later):
http://java.sun.com/javase/technologies/hotspot/gc/memorymanagement_whitepaper.pdf

Having sufficient available memory is the most important factor affecting JVM garbage collection performance.

You can change the Max Heap size by specifying a different value to tibco.env.HEAP_SIZE in deployed engine .tra

file or from Admin GUI.

Other tuning paramters can be used in BW engine by adding the following property to the deployed engine .tra file:
java.extended.properties

For example:
java.extended.properties=-Xincgc  -XX:NewSize=128m

Only resize or fix an internal value (such as NewSize, MaxNewSize ...etc) after careful observation of the system and that observation demonstrated some pathology that you could correct by setting an internal value.