How can we determine if a system resource limitation is causing poor performance for our TIBCO Streaming applications?
TIBCO Streaming performance may be limited by system resources. On Linux, monitor continuous CPU, Disk, Memory, and Network resource usage using the ‘sar’ command from the ‘sysstat’ package.
Ubuntu: sudo apt install sysstat RHEL: sudo yum install sysstat
The ‘sar’ command may be used directly for real-time reporting of current system behavior. If you also want historical behavior logged, enable the ‘sysstat’ service using commands:
systemctl enable sysstat systemctl start sysstat && systemctl status sysstat -l
And to stop:
systemctl stop sysstat && systemctl status sysstat -l
To obtain current system behavior, use command (example):
sar -P ALL -u ALL -b -d -F -I SUM -n TCP,UDP -q -r -S -w -h 1 10 > 10_second_sar.log
The last two parameters are:
To collect and report resource usage continuously until stopped, eliminate the last parameter.
See the ‘man sar’ page for detailed information of these options.
With the above settings, output will look like the following:
The above report includes per-second statistics for the system resources, in order:
CPU usage
Processor Task Switching activity
Processor Interrupts per second
I/O Transfers per second to physical devices
Memory usage
Swap usage
Processor Run Queue size
Block Device I/O Transfers per second
TCP Segments per second
UDP Datagrams per second
Disk usage
Examples of the type of analysis that can be performed with this data are:
Determining a high interrupt rate leading to excessive context switching is slowing overall CPU performance
High Swap use indicates memory is oversubscribed leading to increased latency
High Disk read and write activity is blocking, leading to low performance even though CPUs are not fully utilized
Disk space is limited
Processor Run Queue is frequently greater than 0, indicating there are more active threads than can be effectively task switched and the CPUs are oversubscribed
Network shows bursts of unexpected activity
To use ‘sar’ effectively while troubleshooting TIBCO Streaming, either enable the service to capture system resource statistics continuously and then create historical reports when a problem is observed, or run alongside a TIBCO Streaming process while reproducing a problem in a test environment. To run alongside TIBCO Streaming for troubleshooting we recommend you start these processes in the following order:
1. Start ‘sar’ with output directed to a file.
2. Start the Streaming node using ‘epadmin servicename=A.X start node’.
3. Monitor the engine log until it reports ‘Listening’ as so:
2021-12-03 12:03:21.217000-0500 [20996:Thread- ThreadPool - 1] INFO com.streambase.sb.sbd.net.StreamBaseHTTPServer: sbd at mysystem:44104; pid=20996; version=10.6.1_0a0fdf7f8f3d4f25851d53e0e55c97ce2ece3d22; Listening
4. Start the ‘sbprofile’ command to collect StreamBase engine internal statistics.
5. Run until the problem is observed.
6. Stop the node and ‘sar’ collection (‘sbprofile’ will stop automatically with the node).
Analyze the ‘sbprofile’ and ‘sar’ logs by timestamp together noting system resource usage leading up to the first indications of a problem.