The JVM which your BW engine is running is out of heap memory. Consider tuning the following parameters:
1). JVM heap size.
It is important to set the heap size to a suitable value. It should be set such that it allows the heap to be contained within physical memory to avoid swapping to disk. You can change the Max Heap size by specifying a different value to tibco.env.HEAP_SIZE in the deployed engine .tra file or through the Admin GUI.
2). Turn on MemorySavingMode.
The BW engine currently holds on to each piece of process data through the life of a job, even if it is used once at the beginning and then never reused. Setting the specified property to "true" will force the BW engine to release the process data as soon as it is no longer needed. This mode is off by default. To turn this mode on, set the following property: EnableMemorySavingMode= true .
3). Tuning Max Jobs, Activation Limit and Flow Limit value.
Refer to the BW documentation --> Administration --> Performance Tuning for details. We usually recommend flow limit = 2* max jobs.
If you still get an out of memory exception after you tune the above parameters, you will need to troubleshoot to find out the cause.
1). Check the BW engine log file to see if there are any exceptions other than out of memory.
2). Monitor memory usage.
- On Windows, you can check this using the Task Manager –>Processes Tab. Monitoring the amount of allocated virtual memory size and memory usage allows you to see whether a process is consuming more memory over a given time period.
- On Unix you can run commands like top, prstat, vmstat, ps aux
Sample output of the top command:
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
7664 tibuser 460 1 0 1219M 1071M sleep 46.0H 0.18% tibcoadmin_TIBD
21428 tibuser 999 39 0 1609M 1316M sleep 25:38 0.10% bwengine
28813 tibuser 999 9 0 595M 514M sleep 445:14 0.04% bwengine
20447 tibuser 342 15 0 710M 457M sleep 145:18 0.03% bwengine
27449 tibuser 51 59 0 267M 251M sleep 71:09 0.02% adr3u
27752 tibuser 227 59 0 432M 268M sleep 585:46 0.01% bcengine
You can find the pid of the ActiveMatrix BusinessWorks engine by using ps –ef | grep bwengine .
- You can use Java monitoring tools like JConsole and Visualvm to monitor the memory usage of your BW engine. (Available after JRE 1.5). Refer to KB 35924 on how to use jConsole.
3). Verbose GC output and heap dump.
The command line option -verbose:gc causes information about the heap and garbage collection to be printed at each collection. It has a detailed Java Heap breakdown of each memory space.
Ex: 8109.128: [GC [PSYoungGen: 109884K->14201K(139904K)] 691015K->595332K(1119040K), 0.0454530 secs]
You can refer to the following documentation on how to read a verbose gc log: http://javaeesupportpatterns.blogspot.de/2011/10/verbosegc-output-tutorial-java-7.html
Sun JVM parameters.
java.extended.properties= -verbose:gc -Xloggc:MyGCLog.txt -XX\:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heapdump.log
The -verbose:gc option enables logging of garbage collection (GC) information.
-XX:+PrintGCDetails activate the “detailed” GC logging mode which differs depending on the GC algorithm used.
-XX:+PrintGCTimeStamp add time and date information reflecting the real time passed in seconds since JVM start is added to every line.
-XX:+PrintGCDateStamps each line starts with the absolute date and time when it was written:
-Xloggc:<<file name>> By default the GC log is written to stdout. With -Xloggc:<file> user can specify an output file
IBM JVM parameters.
java.extended.properties=-verbose:gc -Xverbosegclog:/tmp/out1.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
The IBM JVMs dump engine enables HeapDumpOnOutOfMemoryError by default, ensuring that you obtain a heapdump if an OutOfMemoryError occurs. The IBM dump engine provides a number of configuration options using the -Xdump command line options, including disabling the dumps. Refer to IBM document: http://www-01.ibm.com/support/docview.wss?uid=swg21242497#Default_Dump_Agents for details regarding the -Xdump option.
4). Common causes.
- If out of memory happens on a certain period, check the load getting into the BW engine and make sure you have enough resources to handle the load. You need to tune the max jobs flow limit and max heap size to find the best value.
- If out of memory happens on certain data:
. check the message size to make sure the max heap size is enough. One example is the HTTP recevier may receive a large XML payload if incoming message size is not restricted.
. check the process flow, especially error handling to make sure the BW project can process the message correctly. For example, an infinite loop may cause memory exhaustion.
- If memory usage increases gradually over a long period of time and the application reports out of memory, there may be a memory leak. You can also check the verbose GC log for symptons:
. You see frequent Full GC messages in your verbose gc log.
. The post-garbage-collection utilization number is growing over a long period of time.
In case of a memory leak, collect Heap Dump to check which object/class is causing the out of memory problem .
5). Information to send to TIBCO Support to trouble shoot memory issues.
- BW engine log file from engine start till out of memory.
- Deployed engine tra file.
- Verbose GC output from engine start till out of memory.
- Heap dump.
- Details on incoming message rate and message size.
- The BW project.
