Monitor Linux system resources for TIBCO Streaming with sar

Monitor Linux system resources for TIBCO Streaming with sar

book

Article ID: KB0072810

calendar_today

Updated On:

Products

TIBCO Streaming

Description

How can we determine if a system resource limitation is causing poor performance for our TIBCO Streaming applications?

Issue/Introduction

How to run sar to get system resource usage to evaluate performance issues

Environment

Linux operating systems

Resolution

TIBCO Streaming performance may be limited by system resources. On Linux, monitor continuous CPU, Disk, Memory, and Network resource usage using the ‘sar’ command from the ‘sysstat’ package.

 Ubuntu: sudo apt install sysstat RHEL: sudo yum install sysstat

The ‘sar’ command may be used directly for real-time reporting of current system behavior. If you also want historical behavior logged, enable the ‘sysstat’ service using commands:

 systemctl enable sysstat systemctl start sysstat && systemctl status sysstat -l

And to stop:

 systemctl stop sysstat && systemctl status sysstat -l 

To obtain current system behavior, use command (example):

 sar -P ALL -u ALL -b -d -F -I SUM -n TCP,UDP -q -r -S -w -h 1 10 > 10_second_sar.log 

The last two parameters are:

  • 1 – collect data every second
  • 10 – stop and summarize after 10 reports

To collect and report resource usage continuously until stopped, eliminate the last parameter.

See the ‘man sar’ page for detailed information of these options.

With the above settings, output will look like the following:

sar command summary output example

The above report includes per-second statistics for the system resources, in order:

  • CPU usage

  • Processor Task Switching activity

  • Processor Interrupts per second

  • I/O Transfers per second to physical devices

  • Memory usage

  • Swap usage

  • Processor Run Queue size

  • Block Device I/O Transfers per second

  • TCP Segments per second

  • UDP Datagrams per second

  • Disk usage

Examples of the type of analysis that can be performed with this data are:

  • Determining a high interrupt rate leading to excessive context switching is slowing overall CPU performance

  • High Swap use indicates memory is oversubscribed leading to increased latency

  • High Disk read and write activity is blocking, leading to low performance even though CPUs are not fully utilized

  • Disk space is limited

  • Processor Run Queue is frequently greater than 0, indicating there are more active threads than can be effectively task switched and the CPUs are oversubscribed

  • Network shows bursts of unexpected activity

To use ‘sar’ effectively while troubleshooting TIBCO Streaming, either enable the service to capture system resource statistics continuously and then create historical reports when a problem is observed, or run alongside a TIBCO Streaming process while reproducing a problem in a test environment. To run alongside TIBCO Streaming for troubleshooting we recommend you start these processes in the following order:

1. Start ‘sar’ with output directed to a file.

2. Start the Streaming node using ‘epadmin servicename=A.X start node’.

3. Monitor the engine log until it reports ‘Listening’ as so:

 2021-12-03 12:03:21.217000-0500 [20996:Thread- ThreadPool - 1] INFO  com.streambase.sb.sbd.net.StreamBaseHTTPServer: sbd at mysystem:44104; pid=20996; version=10.6.1_0a0fdf7f8f3d4f25851d53e0e55c97ce2ece3d22; Listening

4. Start the ‘sbprofile’ command to collect StreamBase engine internal statistics.

5. Run until the problem is observed.

6. Stop the node and ‘sar’ collection (‘sbprofile’ will stop automatically with the node).

Analyze the ‘sbprofile’ and ‘sar’ logs by timestamp together noting system resource usage leading up to the first indications of a problem.

Additional Information

sbprofile - https://docs.tibco.com/pub/str/10.6.2/doc/html/reference/sbprofile.html#sbprofile%20command

epadmin - https://docs.tibco.com/pub/str/10.6.2/doc/html/rtcmd/index.html