TIBCO ActiveMatrix BusinessWorks 6.x application accumulates and finally run out of memory

TIBCO ActiveMatrix BusinessWorks 6.x application accumulates and finally run out of memory

book

Article ID: KB0094719

calendar_today

Updated On:

Products Versions
TIBCO ActiveMatrix BusinessWorks 6.3.3

Description

There is known memory leak issue in all BW 6.x version prior to TIBCO ActiveMatrix BusinessWorks 6.3.4. Consider the scenario where the main process calls the sub-process and sub-process errors out without gracefully handling that exception or error. In such process design, faulted jobs are kept in memory and application will eventually run out of memory. You can see following log traces in BW appnode log file,

=====================

2016-11-25 18:31:39.593 ERROR [bwEngThread:In-Memory Process Worker-7] com.tibco.pvm.system.PvmSystem - Unexpected Error processing WorkItem
java.lang.OutOfMemoryError: GC overhead limit exceeded
=====================

If you restart the appnode, everything works well for few days until the application gets into the same situation again.

Environment

All version of TIBCO ActiveMatrix BusinessWorks prior 6.3.4

Resolution

How to verify if my case / scenario is same as what is explained in this article?
  • This is not specific to any activity e.g. JDBC or JMS
  • The activity which generates the error or exception e.g. Read File activity generates FileNotFound exception etc. should be in sub-process
  • The error or exception should *NOT* be handled gracefully
  • Please refer attached project. This is a very simple project which has JMS receiver as process starter. Further, it calls the sub-process which always generates fault using 'Throw' activity. At runtime, for every JMS message received, job will be created and be faulted
  • You can make use of OSGi command 'lapi' which retrieves information about active process instances for an application. 'lapi' should lists total jobs running in memory and 'lmetrics' (Print Job metrics for application(s) running on the appnode) commands will show total number of faulted jobs. You can check how many jobs are faulted out of total active jobs.
If your application have already ran out of memory and you have the heapdump, check it for memory leak suspect using tools like IBM MAT. You should see similar result in generated leak suspect report,

Leak Suspects:

==========================
One instance of "com.tibco.pvm.im.rt.xm.util.work.ImxProcessRegistry" loaded by "com.tibco.pvm.im.rt.xm" occupies 885,498,784 (91.31%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>"
==========================

What are the applicable BW6 versions affected by this issue?

All BW6.x version prior to TIBCO ActiveMatrix BusinessWorks 6.3.4

Will this issue be fixed?

Yes, it is fixed in TIBCO ActiveMatrix BusinessWorks 6.3.4.

Is there a workaround for this issue?

Yes. For all versions prior TIBCO ActiveMatrix BusinessWorks 6.3.4, please add below property in appnode's config.ini file, enable OSGi port and restart the appnode.

com.tibco.bx.autoDeleteFailedProcesses=yes

Once appnode is restarted, log into OSGi console and check the output of "getprop com.tibco.bx.autoDeleteFailedProcesses"

You should see the output as
admin@AN1> getprop com.tibco.bx.autoDeleteFailedProcesses=Yes

Issue/Introduction

TIBCO ActiveMatrix BusinessWorks 6.x application accumulates and finally run out of memory

Attachments

TIBCO ActiveMatrix BusinessWorks 6.x application accumulates and finally run out of memory get_app