BWEngine might run into performance issues during high load due to Apache logging. This issue is caused by Apache console logging which is enabled by default. As this logging happens at the console level, it is causing locking issues when more threads are trying to write to the same handler. 

To confirm this issue, please do the following steps.

1. Capture thread dumps when the application is slow/sluggish. Following link has the details of taking thread dump.

Capturing Thread Dump

url : https://docs.tibco.com/pub/activematrix_businessworks/6.4.0/doc/html/GUID-BAC813F8-5DDC-4043-86D6-2F5D6C286B02.html

2. Open the thread dumps and look for the following BLOCKED threads. Existence of these threads confirms that the Apache console handler issue happens in your environment.

"bwEngThread:In-Memory Process Worker-28" #140 prio=5 os_prio=0 tid=0x00007f0358032000 nid=0xa80e waiting for monitor entry [0x00007f04a22e2000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.util.logging.StreamHandler.publish(StreamHandler.java:206)
    - waiting to lock <0x00000000836e67d8> (a java.util.logging.ConsoleHandler)
    at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
    at java.util.logging.Logger.log(Logger.java:738)
    at java.util.logging.Logger.doLog(Logger.java:765)
    at java.util.logging.Logger.logp(Logger.java:931)
    at org.apache.commons.logging.impl.Jdk14Logger.log(Jdk14Logger.java:87)
    at org.apache.commons.logging.impl.Jdk14Logger.debug(Jdk14Logger.java:103)

"Axis2 Task" #510 daemon prio=5 os_prio=0 tid=0x00007f0340026000 nid=0x1772 waiting for monitor entry [0x00007f04887c9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.util.logging.StreamHandler.publish(StreamHandler.java:206)
    - waiting to lock <0x00000000836e67d8> (a java.util.logging.ConsoleHandler)
    at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
    at java.util.logging.Logger.log(Logger.java:738)
    at java.util.logging.Logger.doLog(Logger.java:765)
    at java.util.logging.Logger.logp(Logger.java:931)
    at org.apache.commons.logging.impl.Jdk14Logger.log(Jdk14Logger.java:87)
    at org.apache.commons.logging.impl.Jdk14Logger.trace(Jdk14Logger.java:239)

"bwResourceHTTPClient.IdleConnectionTimeoutThread" #185 daemon prio=5 os_prio=0 tid=0x00007f03a41c0800 nid=0xc309 waiting for monitor entry [0x00007f049b0f1000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.util.logging.StreamHandler.publish(StreamHandler.java:206)
    - waiting to lock <0x00000000836e67d8> (a java.util.logging.ConsoleHandler)
    at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
    at java.util.logging.Logger.log(Logger.java:738)
    at java.util.logging.Logger.doLog(Logger.java:765)
    at java.util.logging.Logger.logp(Logger.java:931)
    at org.apache.commons.logging.impl.Jdk14Logger.log(Jdk14Logger.java:87)
    at org.apache.commons.logging.impl.Jdk14Logger.debug(Jdk14Logger.java:103)
    at org.apache.commons.httpclient.util.IdleConnectionHandler.closeIdleConnections(IdleConnectionHandler.java:108)

org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool.closeIdleConnections(MultiThreadedHttpConnectionManager.java:872)
    - locked <0x0000000088b3e250> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
    at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.closeIdleConnections(MultiThreadedHttpConnectionManager.java:618)
    at org.apache.commons.httpclient.util.IdleConnectionTimeoutThread.handleCloseIdleConnections(IdleConnectionTimeoutThread.java:92)
    at org.apache.commons.httpclient.util.IdleConnectionTimeoutThread.run(IdleConnectionTimeoutThread.java:104)
    - locked <0x0000000088b44008> (a org.apache.commons.httpclient.util.IdleConnectionTimeoutThread)

 

org.apache.commons.logging.Log=org.apache.commons.logging.impl.NoOpLog
​

1. Using appspace/node config.ini file.
   • Stop appnodes.
   • Open the config file and add the following property
     org.apache.commons.logging.Log=org.apache.commons.logging.impl.NoOpLog
   • Start the appnodes.
2. Using appnode tra file
   • Stop appnode.
   • Open the node tra file and append the following property in "java.extended.properties". Please make sure that this property is not commented .
     -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.NoOpLog
   • Start the node