Performance Degradation
Article ID: KB0083677
Updated On:
| Products | Versions |
|---|---|
| TIBCO ActiveMatrix BusinessWorks | 6.x |
Description
BWEngine might run into performance issues during high load due to Apache logging. This issue is caused by Apache console logging which is enabled by default. As this logging happens at the console level, it is causing locking issues when more threads are trying to write to the same handler.
To confirm this issue, please do the following steps.
1. Capture thread dumps when the application is slow/sluggish. Following link has the details of taking thread dump.
Capturing Thread Dump
url : https://docs.tibco.com/pub/activematrix_businessworks/6.4.0/doc/html/GUID-BAC813F8-5DDC-4043-86D6-2F5D6C286B02.html
2. Open the thread dumps and look for the following BLOCKED threads. Existence of these threads confirms that the Apache console handler issue happens in your environment.
"bwEngThread:In-Memory Process Worker-28" #140 prio=5 os_prio=0 tid=0x00007f0358032000 nid=0xa80e waiting for monitor entry [0x00007f04a22e2000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.logging.StreamHandler.publish(StreamHandler.java:206)
- waiting to lock <0x00000000836e67d8> (a java.util.logging.ConsoleHandler)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
at java.util.logging.Logger.log(Logger.java:738)
at java.util.logging.Logger.doLog(Logger.java:765)
at java.util.logging.Logger.logp(Logger.java:931)
at org.apache.commons.logging.impl.Jdk14Logger.log(Jdk14Logger.java:87)
at org.apache.commons.logging.impl.Jdk14Logger.debug(Jdk14Logger.java:103)
"Axis2 Task" #510 daemon prio=5 os_prio=0 tid=0x00007f0340026000 nid=0x1772 waiting for monitor entry [0x00007f04887c9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.logging.StreamHandler.publish(StreamHandler.java:206)
- waiting to lock <0x00000000836e67d8> (a java.util.logging.ConsoleHandler)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
at java.util.logging.Logger.log(Logger.java:738)
at java.util.logging.Logger.doLog(Logger.java:765)
at java.util.logging.Logger.logp(Logger.java:931)
at org.apache.commons.logging.impl.Jdk14Logger.log(Jdk14Logger.java:87)
at org.apache.commons.logging.impl.Jdk14Logger.trace(Jdk14Logger.java:239)
"bwResourceHTTPClient.IdleConnectionTimeoutThread" #185 daemon prio=5 os_prio=0 tid=0x00007f03a41c0800 nid=0xc309 waiting for monitor entry [0x00007f049b0f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.logging.StreamHandler.publish(StreamHandler.java:206)
- waiting to lock <0x00000000836e67d8> (a java.util.logging.ConsoleHandler)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
at java.util.logging.Logger.log(Logger.java:738)
at java.util.logging.Logger.doLog(Logger.java:765)
at java.util.logging.Logger.logp(Logger.java:931)
at org.apache.commons.logging.impl.Jdk14Logger.log(Jdk14Logger.java:87)
at org.apache.commons.logging.impl.Jdk14Logger.debug(Jdk14Logger.java:103)
at org.apache.commons.httpclient.util.IdleConnectionHandler.closeIdleConnections(IdleConnectionHandler.java:108)
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool.closeIdleConnections(MultiThreadedHttpConnectionManager.java:872)
- locked <0x0000000088b3e250> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.closeIdleConnections(MultiThreadedHttpConnectionManager.java:618)
at org.apache.commons.httpclient.util.IdleConnectionTimeoutThread.handleCloseIdleConnections(IdleConnectionTimeoutThread.java:92)
at org.apache.commons.httpclient.util.IdleConnectionTimeoutThread.run(IdleConnectionTimeoutThread.java:104)
- locked <0x0000000088b44008> (a org.apache.commons.httpclient.util.IdleConnectionTimeoutThread)
To confirm this issue, please do the following steps.
1. Capture thread dumps when the application is slow/sluggish. Following link has the details of taking thread dump.
Capturing Thread Dump
url : https://docs.tibco.com/pub/activematrix_businessworks/6.4.0/doc/html/GUID-BAC813F8-5DDC-4043-86D6-2F5D6C286B02.html
2. Open the thread dumps and look for the following BLOCKED threads. Existence of these threads confirms that the Apache console handler issue happens in your environment.
"bwEngThread:In-Memory Process Worker-28" #140 prio=5 os_prio=0 tid=0x00007f0358032000 nid=0xa80e waiting for monitor entry [0x00007f04a22e2000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.logging.StreamHandler.publish(StreamHandler.java:206)
- waiting to lock <0x00000000836e67d8> (a java.util.logging.ConsoleHandler)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
at java.util.logging.Logger.log(Logger.java:738)
at java.util.logging.Logger.doLog(Logger.java:765)
at java.util.logging.Logger.logp(Logger.java:931)
at org.apache.commons.logging.impl.Jdk14Logger.log(Jdk14Logger.java:87)
at org.apache.commons.logging.impl.Jdk14Logger.debug(Jdk14Logger.java:103)
"Axis2 Task" #510 daemon prio=5 os_prio=0 tid=0x00007f0340026000 nid=0x1772 waiting for monitor entry [0x00007f04887c9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.logging.StreamHandler.publish(StreamHandler.java:206)
- waiting to lock <0x00000000836e67d8> (a java.util.logging.ConsoleHandler)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
at java.util.logging.Logger.log(Logger.java:738)
at java.util.logging.Logger.doLog(Logger.java:765)
at java.util.logging.Logger.logp(Logger.java:931)
at org.apache.commons.logging.impl.Jdk14Logger.log(Jdk14Logger.java:87)
at org.apache.commons.logging.impl.Jdk14Logger.trace(Jdk14Logger.java:239)
"bwResourceHTTPClient.IdleConnectionTimeoutThread" #185 daemon prio=5 os_prio=0 tid=0x00007f03a41c0800 nid=0xc309 waiting for monitor entry [0x00007f049b0f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.logging.StreamHandler.publish(StreamHandler.java:206)
- waiting to lock <0x00000000836e67d8> (a java.util.logging.ConsoleHandler)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
at java.util.logging.Logger.log(Logger.java:738)
at java.util.logging.Logger.doLog(Logger.java:765)
at java.util.logging.Logger.logp(Logger.java:931)
at org.apache.commons.logging.impl.Jdk14Logger.log(Jdk14Logger.java:87)
at org.apache.commons.logging.impl.Jdk14Logger.debug(Jdk14Logger.java:103)
at org.apache.commons.httpclient.util.IdleConnectionHandler.closeIdleConnections(IdleConnectionHandler.java:108)
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool.closeIdleConnections(MultiThreadedHttpConnectionManager.java:872)
- locked <0x0000000088b3e250> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.closeIdleConnections(MultiThreadedHttpConnectionManager.java:618)
at org.apache.commons.httpclient.util.IdleConnectionTimeoutThread.handleCloseIdleConnections(IdleConnectionTimeoutThread.java:92)
at org.apache.commons.httpclient.util.IdleConnectionTimeoutThread.run(IdleConnectionTimeoutThread.java:104)
- locked <0x0000000088b44008> (a org.apache.commons.httpclient.util.IdleConnectionTimeoutThread)
Issue/Introduction
Engine throughput is degraded under high load
Environment
All
Resolution
Disabling apache logging would resolve this issue. Following property can be used to achieve that.
org.apache.commons.logging.Log=org.apache.commons.logging.impl.NoOpLog
This property can be set as follows.
- Using appspace/node config.ini file.
- Stop appnodes.
- Open the config file and add the following property
org.apache.commons.logging.Log=org.apache.commons.logging.impl.NoOpLog - Start the appnodes.
- Using appnode tra file
- Stop appnode.
- Open the node tra file and append the following property in "java.extended.properties". Please make sure that this property is not commented .
-Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.NoOpLog - Start the node
Feedback
Yes
No
Powered by
