What are the recommended Production settings for TIBCO Streaming to make troubleshooting easier? 

Guidance how to add logging to help tune a Streaming server.

The main issue administrators encounter with long running TIBCO® Streaming applications is with memory use and the problems related to pauses for garbage collection when running low on Heap memory.

The default recommended settings from the documentation section:
  TIBCO Streaming > StreamBase Admin Guide > Tuning StreamBase Applications > Java VM Memory Settings,
  JVM Memory for EventFlow Fragments
are:

configuration = {
  StreamBaseEngine = {
    jvmArgs = [
      "-Xmx2048m"
      "-Xms2048m"
      "-XX:+UseG1GC"
      "-XX:MaxGCPauseMillis=500"
      "-XX:ConcGCThreads=1"
    ]

The -Xmx sets the maximum heap. And -Xms sets the pre-allocated memory. We recommend these be set the same to keep the garbage collector from evaluating whether to request more memory from the OS. Examine heap use and garbage collector behavior under production-level loads to determine an appropriate value to replace the "2048m" (two gigabyte) value in this example.

The 500 millisecond target works well for most Streaming applications. Setting this to a lower (shorter) value may have the negative effect of giving the garbage collector too little time to recover as much memory as it could, leading to slower operation later.

If your application is taking greater than 10% of the CPU time doing garbage collection, increase the ConcGCThreads setting to a larger value up to 8 threads and no more than half of the physical CPUs available on the host system.

To add detailed logging of JVM garbage collection activity, add these jvmArgs settings:

"-verbose:gc"
"-Xlog:gc:gc.log"
"-XX:+PrintVMOptions"

These create a gc.log file in the node directory and this file is included when creating a node snapshot for troubleshooting. Create a node snapshot ZIP file using command:
  epadmin –servicename={nodename}.{clustername} create snapshot
The gc.log file shows the type, frequency, and effectiveness of each garbage collection event. There are many online guides and utilities for how to interpret this log. In a well running Java application, garbage collection is able to regularly return the Heap use to a consistent lower boundary.

If you are running a LiveView server (LDMEngine) in the node, make sure the minimum -Xmx setting is “4g”, and add the following system properties outside the jvmArgs section:

systemProperties = {
  "liveview.lvmonitor.enable.os.cmd" = "true"
  "liveview.lvmonitor.enable.profiler" = "true"
  "liveview.lvmonitor.profile.oldest.hours" = "48"
}

The 'os.cmd' option turns on use of 'sar' on Linux systems to report to the engine log system resource performance metrics. The 'profile' options enable the integrated 'sbprofile' process which logs to the node lv-profile/ directory. The old profile logs are removed if they are older than 48 hours to keep from using excessive disk space. These logs are included in the node snapshot report.

The same profile logging may be enabled by running the ‘sbprofile’ command and connecting to a running server as described in the documentation here:
  TIBCO Streaming > StreamBase References > Command Reference > sbprofile

Also, see the topics in section:
  TIBCO Streaming > Test/Debug Guide > Profiling Operator Performance

JVM Memory for EventFlow Fragments
sbprofile
Profiling Operator Performance