LiveView Server warnings indicating that the computer system is overloaded

LiveView Server warnings indicating that the computer system is overloaded

book

Article ID: KB0076049

calendar_today

Updated On:

Products Versions
TIBCO Live Datamart 7.x

Description

The following types of errors indicate that the computer system is under-powered and oversubscribed for the current CPU, Memory, Disk, or Network demand from all running applications, including the LiveView server. They indicate lack of sufficient system resources to run the current LiveView application:
 
WARN LVMonitor - Prolonged JVM stutter: Total time 110.6 seconds

WARN LVMonitor - Low Available Heap: Max size 20,446 MB, Free 279 MB

WARN  c.s.s.r.m.q.StreamTargetSequencer - Queue: "Alerts.EngineFactory.EngineImpl:1:ParallelSequence" expanding by 262144 tuples 
because the consumer is slower than the producer. Waited up to 1000 milliseconds (actually 5.35728332579178E8). 

WARN com.streambase.liveview.server.table.StreamBaseConnectionPool - StreamBaseConnection: Exiting on error
com.streambase.sb.StreamBaseException: java.net.SocketTimeoutException: Configured server quiescent time of 120000 MS exceeded 
to: localhost/127.0.0.1:10000

How can we learn more about what is causing the problem?

Issue/Introduction

Troubleshooting guidance

Resolution

These typically indicate that there is too little memory allocated or available to the LiveView Server, but may also indicate that there are too many active processes for the available CPU threads available to efficiently process the available data. They indicate the presence of high update latency, higher memory use, and possible network disconnects.

To troubleshoot these and related issues Support may request certain additional settings to be added to the server configuration. Reproducing the issue in a test environment allows additional instrumentation to be used which should not be used in a production environment.

If sufficient system resources become available, the LiveView Server process is expected to fully recover and be able to process data normally.

Recommended Settings for Troubleshooting


1. Enable profile logging and OS performance logging

By default the profile log will roll every ~50 MB and profiles older than 24 hours are deleted. The profiles are kept in ${PROJECTDIR}/lv-profiles. For TIBCO Streaming 10, this location is reported in console message:
INFO  com.streambase.liveview.config.LVSBConf: Project output folder is {path-to}\target\liveview

The profile logs report all self-monitored LiveView Server statistics including memory use and CPU use.

The OS logs (on Linux based systems) will be written to the normal engine log and will show whether there is any OS-level process swapping introducing delays, as well as CPU use, network use, and related operating-system metrics. These log messages will be written to the same console output as other log messages. Please ensure your logging configuration does not discard these messages.

For StreamBase version 7, the additional sbd.sbconf settings to enable these logs are:
<java-vm>
    <sysproperty name="liveview.lvmonitor.enable.profiler" value="true"/>
    <sysproperty name="liveview.lvmonitor.enable.os.cmd" value="true"/>
...

For TIBCO Streaming version 10, the additional HOCON configuration settings are:
configuration = {
    LDMEngine = {
        systemProperties = {
          "liveview.lvmonitor.enable.profiler" = "true"
          "liveview.lvmonitor.enable.os.cmd" = "true"
        }   
...
These settings are also recommended for Production use to help perform root-cause analysis of problems impacting Production.


2. Collect full console logs

When reproducing a problem during troubleshooting:
  • Do not use a custom logging configuration. 
  • Do reproduce the problem in a testing environment. 
  • Do run as a normal user and normal application and not a Windows Service.
In Production, make sure any custom logging configuration preserves console logs for each session. When providing console and profile logs to Support, make sure they cover the same time-frames.


3. Turn on JVM garbage-collection performance logging

Garbage collection logs are useful for diagnosing problems related specifically to memory use. If the Java VM is taking too much CPU time to perform garbage collection, this can increase LiveView Server latency and internal queuing. This is often caused by setting the JVM Heap size too small for the amount of data to be preserved in LiveView tables.

For TIBCO StreamBase version 7, add to the sbd.sbconf:
<java-vm>
    <param name="jvm-args" value=" 
    -verbose:gc 
    -Xloggc:./gc.log 
    -XX:+PrintGCDetails 
    -XX:+PrintGCDateStamps
    "/>
...

The equivalent settings for TIBCO Streaming 10 are:
configuration = {
  StreamBaseEngine = {
    jvmArgs = [
      "-Xmx2048m"
      "-Xms512m"
      "-XX:+UseG1GC"
      "-XX:MaxGCPauseMillis=500"
      "-XX:ConcGCThreads=1"
    ]
  }
...


4. Report installation and machine capabilities

For TIBCO StreamBase 7, run a StreamBase health report from each server machine. This will provide install and OS-level details to assist troubleshooting.

To generate the report, run:
    sbhealth > sbhealth.log 

For TIBCO Streaming 10, create a node snapshot report using command:
    epadmin servicename=A.X create snapshot destination=.

Review these and provide them to TIBCO Support.


5. Describe the use-case and application architecture

Communicate to Support as many details as you can regarding:
  • Expected data rates (What are the publish rate going into the LV tables? How often are new queries sent from client applications, and what queries are those?)
  • Where is the data coming from? (For example, what StreamBase adapters are being used in your publishing application?)
  • How wide are the LV tables? (Put another way, How many columns?)

This will greatly help us diagnose any performance issues.