How to get a thread-dump from a TIBCO Streaming Node
Article ID: KB0073006
Updated On:
| Products | Versions |
|---|---|
| TIBCO Streaming | 10.x and later |
Description
How can an administrator or developer get a Java-level thread-dump from the applications in a TIBCO Streaming node?
Environment
All
Resolution
A TIBCO Streaming Node may have one or more application Engines (StreamBase, LiveView, and Java), each in their own Java Virtual Machine (JVM) instances. Several thread-dump reports from each engine may be necessary to troubleshoot an issue.
When troubleshooting an application hang it is useful to get a Java-level thread-dump of process stack traces. The thread-dump shows each thread's stack-trace which identifies the last-called method on which the thread may be blocked. Additionally, this can identify deadlocks between multiple threads. A TIBCO Streaming application has usefully named threads, class-names, and methods, so specific adapters can be identified in the stack traces.
TIBCO Support may request thread-dump reports to troubleshoot a reported issue.
Identify the Process ID (PID) of the application engine process by viewing the application server log found in the node "logs/" folder. Within a node there may be one or more engines. Each engine is its own Java Virtual Machine (JVM) with its own Process ID and server log. The typical name for an engine log is "default-engine-for-{package}.{project-name}.log".
For example:
default-engine-for-com.example.chirp.log
The engine name may be overridden by the NodeDeploy configuration, "engines" element. The log-file name may be overridden by a custom logback configuration.
The "Listening" line in each log reports the PID, as so:
You may also be able to identify the engine from the JVM's reported by the 'jps' command.
Example:
C:\>jps
In this case, connect to and collect thread-dumps from process '3556'. The '19700' process is a different instance of the same application launched from SB Studio.
2. Collect a thread dump.
The 'jstack' command is included in with the Oracle Java JDK installed within the TIBCO Streaming distribution, at:
tibco/sb-cep/10.6/jdk/bin/jstack
It is strongly encouraged that you use the 'jstack' implementation from the same JDK which Streaming uses to run the application. The 'jstack' from other Java versions may not be entirely compatible and miss collecting some information from the engine process.
Example:
A. The last call was to 'operator.sendOutput()' which indicates the operator or adapter cannot emit a tuple because the module thread is blocked on some other activity. In a correctly running application the 'sendOutput()' call is resolved quickly.
B. The last call was into an adapter 3rd-party API method (such as: connect(), get(), receive()) which is not returning. Blocking calls may legitimately block for long periods of time if there is no data available. Most calls like this also incorporate a timeout so the call does not block indefinitely.
C. The thread-stack shows a call to shutdown() resulting in a non-functional operator or adapter.
D. The thread is blocked on a resource held by another thread, which may be blocked for its own reason. This may also be a Java deadlock where multiple resources are held by multiple threads in a cycle.
When troubleshooting an application hang it is useful to get a Java-level thread-dump of process stack traces. The thread-dump shows each thread's stack-trace which identifies the last-called method on which the thread may be blocked. Additionally, this can identify deadlocks between multiple threads. A TIBCO Streaming application has usefully named threads, class-names, and methods, so specific adapters can be identified in the stack traces.
TIBCO Support may request thread-dump reports to troubleshoot a reported issue.
Steps to Get a Java Thread-Dump
1. Get the Process ID of the application engine.Identify the Process ID (PID) of the application engine process by viewing the application server log found in the node "logs/" folder. Within a node there may be one or more engines. Each engine is its own Java Virtual Machine (JVM) with its own Process ID and server log. The typical name for an engine log is "default-engine-for-{package}.{project-name}.log".
For example:
default-engine-for-com.example.chirp.log
The engine name may be overridden by the NodeDeploy configuration, "engines" element. The log-file name may be overridden by a custom logback configuration.
The "Listening" line in each log reports the PID, as so:
2021-09-28 10:34:38.426000-0400 [3556:Thread- ThreadPool - 1] INFO com.streambase.sb.sbd.net.StreamBaseHTTPServer: sbd at mysystem:38046; pid=3556; version=10.6.1_...; Listeningand most lines in the log will begin with a timestamp, PID, and a thread name, as so:
2021-09-28 10:44:40.003000-0400 [3556:AsyncLogger] INFO ...In both cases from this example, the PID is '3556'.
You may also be able to identify the engine from the JVM's reported by the 'jps' command.
Example:
C:\>jps
| 12996 | DtmEngine.exe, a Streaming node service (not a node application) |
| 19700 com_example_chirp_chirp0 | the application engine launched from SB Studio |
| 7092 Jps | the 'jps' command itself |
|
17928 org.eclipse.equinox.launcher | TIBCO Streaming Studio in eclipse |
| 3556 chirp | the application engine JVM as run from the command-line |
2. Collect a thread dump.
The 'jstack' command is included in with the Oracle Java JDK installed within the TIBCO Streaming distribution, at:
tibco/sb-cep/10.6/jdk/bin/jstack
It is strongly encouraged that you use the 'jstack' implementation from the same JDK which Streaming uses to run the application. The 'jstack' from other Java versions may not be entirely compatible and miss collecting some information from the engine process.
Example:
$ /opt/tibco/sb-cep/10.6/jdk/bin/jstack 3556 > jstack_1.log $ sleep 10 $ /opt/tibco/sb-cep/10.6/jdk/bin/jstack 3556 > jstack_2.log ...
Analyzing Streaming Thread-Dumps
Take several thread-dump reports in sequence at about 10 second intervals. Compare how the stack traces change from report to report. If a stack-trace for a thread is not changing it may not have received input that would make it become active, but it may also be blocked. Here are several examples:A. The last call was to 'operator.sendOutput()' which indicates the operator or adapter cannot emit a tuple because the module thread is blocked on some other activity. In a correctly running application the 'sendOutput()' call is resolved quickly.
B. The last call was into an adapter 3rd-party API method (such as: connect(), get(), receive()) which is not returning. Blocking calls may legitimately block for long periods of time if there is no data available. Most calls like this also incorporate a timeout so the call does not block indefinitely.
C. The thread-stack shows a call to shutdown() resulting in a non-functional operator or adapter.
D. The thread is blocked on a resource held by another thread, which may be blocked for its own reason. This may also be a Java deadlock where multiple resources are held by multiple threads in a cycle.
Issue/Introduction
Use JDK 'jstack' command to obtain thread-dump reports from each Streaming engine's JVM.
Feedback
Yes
No
Powered by
