TIBCO Streaming runs in a Java Virtual Machine (JVM) and the JVM may crash when an uncatchable error occurs in a native library in code not managed by Java. In these cases, the application is stopped and the Linux operating system crash handler is started. Java will attempt to create in the Streaming node directory an hs_err_pid.log named after the Process ID of the JVM instance which crashed (for example, hs_err_pid3977817.log), but this may fail to be created if the error stops all JVM processing.

Provide to Support:

1. a node snaphot,
2. an hs_err_pid*.log,
3. and a gdb.txt file

To create a node snapshot of a fully crashed node, use command:
  epadmin create snapshot installpath={path}

If created, the hs_err_pid.log file will report the location the OS will try to create the core file. It will look like this (example):

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f9db254d190, pid=3977817, tid=61484
#
# JRE version: Java(TM) SE Runtime Environment 18.9 (11.0.7+8) (build 11.0.7+8-LTS)
# Java VM: Java HotSpot(TM) 64-Bit Server VM 18.9 (11.0.7+8-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libtib.so.1+0x58190]  _tibFormat_GetFieldIndex+0x40
#
# Core dump will be written. Default location: Core dumps may be processed with 
  "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to 
  /opt/app/data/project/instance/A.businessprocess_app_1/core.3977817)
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

If permitted, the Linux operating system will also create a core file in the node directory as well (for example core.3977817, also named after the JVM PID). The OS may be prevented from creating the core file if disk space it limited or if the user limit ulimit -c (in blocks, typically 4KB in size) shows a value smaller than the Heap size for the Streaming process JVM. The Streaming Process JVM maximum heap size is set with JVM parameter -Xmx{size} set in the StreamBaseEngine jvm-args HOCON section. To make sure a core file is created for any future crash, have an administrator set system-wide ulimit -c unlimited.

Once a core file is created, then on the same Linux system, run 'gbd {program} {corefile}' (gdb is the GNU Debugger). For a Streaming application, the "program" is the Streaming server JVM process (example):  /opt/tibco/sb/10.6.1/distrib/kabira/component/DtmEngine/prodbin/DtmEngine.

The gdb application must be run on the same Linux system on which crash occurred in order to have the same set of libraries, library versions, and filesystem locations as was present when the core file was created. If the libraries are changed, or the core file has been moved to a different system, it is uninterpretable. Do not provide the core file directly to Support unless specifically requested.

The gdb command would be (example):

gdb /opt/tibco/sb/10.6.1/distrib/kabira/component/DtmEngine/prodbin/DtmEngine {core file}

In gdb, run these commands, which will write the output to gdb.txt in the current directory:

gdb>set logging enabled
gdb>bt
gdb>where
gdb>thread apply all bt
gdb>quit

Command legend:
  set logging enabled: append all gdb input and output to the gdb.txt file.
  bt: To print a backtrace of the entire stack, use the backtrace command, or its alias bt . This command will print one line per frame for frames in the stack. By default, all stack frames are printed. The thread apply all bt variant prints the backtrace for all threads in the dump.
  where: Shows current line number and which function the crash occurred at.

If you have any difficulty with the above commands, contact Support for assistance.