BE application uses too much CPU

Products	Versions
TIBCO BusinessEvents Enterprise Edition	5.6 and later

Description

My BE application is using an unexpectedly high percentage of CPU. How can I troubleshoot further to reduce the CPU usage?

Resolution

1. Identify the high cpu-consuming threads with OS-level tools like top (Linux) and Process Explorer (Windows).

Linux example:

$ top -H

top - 12:32:23 up  1:41,  0 users,  load average: 1.30, 0.81, 0.78
Threads: 222 total,   1 running, 205 sleeping,   0 stopped,  16 zombie
%Cpu(s):  6.1 us,  4.0 sy,  0.0 ni, 88.1 id,  0.1 wa,  0.0 hi,  1.6 si,  0.0 st
MiB Mem :  12545.7 total,   1367.1 free,   4477.8 used,   6700.8 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.   7497.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6814 sbadmin   20   0 5343564 934540  29908 S  10.3   7.3   0:09.95 /Channels/EMS/B
   6275 sbadmin   20   0 1421524  44612   9736 S   9.6   0.3   0:07.90 tibemsd
   6902 sbadmin   20   0 7653216 641284  29972 S   7.0   5.0   0:02.66 sbd-java
   6811 sbadmin   20   0 5343564 934540  29908 S   3.3   7.3   0:03.18 TIBCO EMS TCPLi
   ...

In the above example, the offending thread is PID 6814.

2. Convert the PID value from decimal to hexadecimal format using a free online tool. For example:

https://www.rapidtables.com/convert/number/decimal-to-hex.html

In this case, the hex value for the decimal PID 6814 is 1A9E.

3. Collect a thread dump using the jstack or jcmd tools provided with your TIBCO JRE installation (under $TIBCO_HOME/tibcojre64/<version>/bin/):

$TIBCO_HOME/tibcojre64/11/bin/jstack 6814 > jstack1.txt

or..

$TIBCO_HOME/tibcojre64/11/bin/jcmd 6814 Thread.print > jstack1.txt

4. Inspect the resulting thread dump (i.e. jstack1.txt), and search for the hex value obtained in step 2 above. Below is the stack trace for the thread whose hex ID value is 1A9E:

"/Channels/EMS/BookDataDestination-1" #38 daemon prio=5 os_prio=0 cpu=17660.03ms elapsed=163.62s tid=0x00000000037a0800 nid=0x1a9e runnable  [0x00007fca8d3a7000]
   java.lang.Thread.State: RUNNABLE
    at java.io.FileOutputStream.writeBytes(java.base@11.0.9/Native Method)
    at java.io.FileOutputStream.write(java.base@11.0.9/FileOutputStream.java:354)
    at java.io.BufferedOutputStream.write(java.base@11.0.9/BufferedOutputStream.java:123)
    - locked <0x00000000c1665a90> (a java.io.BufferedOutputStream)
        ...

In this case, we can see that the offending thread relates to the EMS channel activity for the destination 'BookDataDestination'.

5. To improve performance, review the settings for this destination configured in the project's CDD file:

<agent-classes>
        <inference-agent-class id="inference-class">
            <rules>
                <ref>all-rules</ref>
            </rules>
            <destinations>
                <destination id="BookDataDestination_5C9EC3CD">
                    <pre-processor/>
                    <queue-size/>
                    <thread-count/>
                    <threading-model>shared-queue</threading-model>
                    <thread-affinity-rule-function/>
                    <uri>/Channels/EMS/BookDataDestination</uri>
                </destination>
            </destinations>
            <startup/>
            <shutdown/>
            <local-cache>
                <eviction>
                    <max-size/>
                    <max-time/>
                </eviction>
            </local-cache>
            <shared-queue>
                <size>1024</size>
                <workers>10</workers>
            </shared-queue>

Here, the number of worker threads is set to 10 for this shared queue model. To reduce the CPU overall usage relating to this destination, increase the worker threads from the default value of 10.

Issue/Introduction

Outlines steps for identifying high cpu-consuming threads in a running BE application.

Welcome to "KB Articles"