Description:
With a high message inflow rate on the bwpm.event queue, and if the heap size is set to the Tomcat server is not enough for the TIBCO BusinessWorks(TM) ProcessMonitor(BWPM) server to process those messages, the  server will run out of memory and throw the error,  "java.lang.OutOfMemoryError: Java heap space" in  the TIBCO BusinessWorks(TM) ProcessMonitor  server.log. Once this error occurs, the messages from  the bwpm.event queues will not be consumed unless restarting the Tomcat server.  After some time, when the TIBCO BusinessWorks(TM) ProcessMonitor server runs out of memory, you will see messages on the bwpm.queue are backed up and are not being consumed.

Symptoms:
TIBCO BusinessWorks(TM) ProcessMonitor(BWPM) server throws the error,  "java.lang.OutOfMemoryError: Java heap space" in the server.log.  In the server.log file, you will see errors resembling the following:
******************************
2014-06-06 17:25:01,001 ERROR [LOCAL_Worker-1] org.quartz.core.JobRunShell: Job LOCAL.NJAMS_JOB_UPDATE_DATA_PROVIDER_STATISTICS threw an unhandled Exception:


java.lang.IllegalStateException: Hazelcast Instance is not active!
    at com.hazelcast.impl.FactoryImpl.initialChecks(FactoryImpl.java:728)
    at com.hazelcast.impl.MProxyImpl.beforeCall(MProxyImpl.java:102)
    at com.hazelcast.impl.MProxyImpl.get(MProxyImpl.java:112)
    at com.faizsiegeln.njams.dataprovider.DataProviderStatistics.getDataProviderStatistics(nJAMS:37)
    at com.faizsiegeln.njams.jobs.JobDataProviderStatistics.execute(nJAMS:25)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)

' 2014-06-09 06:13:35,719 ERROR [hz._hzInstance_1_njamsGroup.cached.thread-1253] com.faizsiegeln.njams.d.a: msgProcessor: msg. processing failed
com.faizsiegeln.njams.a.a: Java heap space'
******************************

 After this issue occurs you will see messages on the bwpm.queue being backed up and not being consumed. Once restarting the Tomcat server, TIBCO BusinessWorks(TM) ProcessMonitor will start consuming messages. After some time, when the TIBCO BusinessWorks(TM) ProcessMonitor server runs out of memory, you will see messages on the bwpm.queue are backed up and not being consumed.
Cause:
Tomcat has run out of memory due to not enough heap size. The max heap size of Tomcat must be set to at least, 500MB + <sum of all data provider threads> * 100MB. If you set it to a lower value than this the issue will occur while processing messages from the bwpm.event queues, the data provider will run out of memory.