Shared memory becomes exhausted due to disk-based query table operations

Shared memory becomes exhausted due to disk-based query table operations

book

Article ID: KB0071631

calendar_today

Updated On:

Products Versions
TIBCO Streaming 10.6 and later

Description

In some cases, a Streaming application needs to, upon startup, perform a large number of update and/or insert operations on a disk-based query table. This may quickly exhaust the shared memory configured for the application (which is 512MB by default). The resulting console log will show warnings about this shortly before the application exits:
 
INFO  com.streambase.sb.sbd.net.StreamBaseHTTPServer: sbd at jmulcahe-X1E:54430; pid=23880; version=10.6.3_258554276e2ca4096895318524265e1220d38934; Listening
WARN  com.tibco.ep.dtm.sharedmemory: (smalloc.cpp:811) Low memory warning: 89 % of shared memory in use for segment 2
WARN  com.tibco.ep.dtm.sharedmemory: (smalloc.cpp:811) Low memory warning: 80 % of shared memory in use for segment 3
WARN  com.tibco.ep.dtm.sharedmemory: (smalloc.cpp:811) Low memory warning: 80 % of shared memory in use for segment 1
WARN  com.tibco.ep.dtm.sharedmemory: (smalloc.cpp:811) Low memory warning: 80 % of shared memory in use for segment 4
WARN  com.tibco.ep.dtm.sharedmemory: (smalloc.cpp:811) Low memory warning: 80 % of shared memory in use for segment 5
WARN  com.tibco.ep.dtm.sharedmemory: (smalloc.cpp:811) Low memory warning: 80 % of shared memory in use for segment 0
WARN  com.tibco.ep.dtm.sharedmemory: (smalloc.cpp:811) Low memory warning: 80 % of shared memory in use for segment 6

This can especially occur in migrated applications from (StreamBase 7 and earlier), as shared memory was not used prior to Streaming 10. 

Issue/Introduction

This article highlights a particular use case where a Streaming application may exhaust shared memory due to disk-based query table operations.

Resolution

Consider the following resolution options:

1. Increase the shared memory allowed for this application, so long as there is sufficient disk space available on the machine. You can increase the shared memory for your application by using a Runtime Application HOON configuration. For example..
 
name = "sample-ApplicationDefinition-document"
type = "com.tibco.ep.dtm.configuration.application"
version = "1.0.0"
configuration = {
    ApplicationDefinition = {
        execution = {
            nodeTypes = {
                mytype1 = {
                    sharedMemory = {
                        memorySizeBytes = "3G"    
                    }    
                }    
            }    
        }
    }
}

The above example increases shared memory to 3GB.

This is discussed in more detail in the Help under Home > Configuration Guide > Runtime Configuration Types > Runtime Application Configuration.

2. Stop using disk-based query tables, and configure them to use transactional memory instead. This will again require an increase in shared memory. However, you get the benefit of having the data automatically shared between nodes. This is discussed in more detail in the Help under Home > Authoring Guide > Using Query Tables > Persistence of Query Tables > (sub-heading) Transactional Memory Option.