How to correct Streaming node ossm requested size is larger than maximum allocation size

How to correct Streaming node ossm requested size is larger than maximum allocation size

book

Article ID: KB0070619

calendar_today

Updated On:

Products Versions
TIBCO Streaming 10.x, 11.x, and later

Description

After running for a little while the application fails with this error:

2024-01-01 15:14:13.012000+0000 [7108:10252] ERROR com.tibco.ep.dtm.sharedmemory: (smalloc.cpp:566) sMMalloc: allocation failed for 67130240 
bytes from C:...\{nodename}\ossm: requested size is larger than maximum allocation size of 67130216. file osproxy.cpp, line 4099
2024-01-01 15:14:13.012000+0000 [7108:10252] WARN  com.tibco.ep.dtm.runtime: (osdisp.cpp:1342) CATCH(DSEException)
2024-01-01 15:14:13.012000+0000 [7108:10252] WARN  com.tibco.ep.dtm.runtime: (dseexcpt.cpp:405) STACK TRACE:
  +++ START +++
    smalloc.cpp (568): THROW SWSMAlloc::SMAllocResourceError
    smalloc.cpp (711): RETHROW SWSMAlloc::SMAllocResourceError
    osproxy.cpp (4131): RETHROW SWSMAlloc::SMAllocResourceError
    osproxy.cpp (4022): RETHROW SWSMAlloc::SMAllocResourceError
    switchadmin_acc0.cpp (2401): RETHROW SWSMAlloc::SMAllocResourceError
    switchadmin_mimpl1.cpp (198): RETHROW SWSMAlloc::SMAllocResourceError
    switchadmin_mimpl1.cpp (200): RETHROW SWSMAlloc::SMAllocResourceError
    switchadmin_stb1.cpp (2749): RETHROW SWSMAlloc::SMAllocResourceError
    switchadmin_stb1.cpp (2751): RETHROW SWSMAlloc::SMAllocResourceError
    switchadmin_stb1.cpp (3623): RETHROW SWSMAlloc::SMAllocResourceError
    historyplugin_stb0.cpp (4392): RETHROW SWSMAlloc::SMAllocResourceError
    historyplugin_mimpl0.cpp (752): RETHROW SWSMAlloc::SMAllocResourceError
    historyplugin_mimpl0.cpp (754): RETHROW SWSMAlloc::SMAllocResourceError
    swtimer_stb0.cpp (1866): RETHROW SWSMAlloc::SMAllocResourceError
    swtimer_stb0.cpp (1060): RETHROW SWSMAlloc::SMAllocResourceError
    swtimer_stb0.cpp (1062): RETHROW SWSMAlloc::SMAllocResourceError
    swtimer_stb0.cpp (1795): RETHROW SWSMAlloc::SMAllocResourceError
    swtimer_mimpl0.cpp (1273): RETHROW SWSMAlloc::SMAllocResourceError
    swtimer_mimpl0.cpp (1275): RETHROW SWSMAlloc::SMAllocResourceError
    swtimer_skl0.cpp (1350): RETHROW SWSMAlloc::SMAllocResourceError
    osdisp.cpp (1342): DUMP SWSMAlloc::SMAllocResourceError
  +++ END +++

Resolution

This error is related to to the Transactional Memory configuration for the node which is configured too small for size of individual items stored by the running application. Transactional Memory is native memory separate from the JVM Heap. The maximum size of a single object or Query Table row or tuple is related to the overall configured size according to the following table:

Maximum Object Size Based On Shared Memory Size

Shared Memory SizeMax Object Size
512MB (default)32MB
1024MB (1GB)64MB
2048MB (2GB)128MB
4096MB (4GB)256MB

The greatest object size permitted is 1024MB (1GB) when the Shared Memory Size is 16GB or larger.

See the "sharedMemory" setting in the ApplicationDefinition HOCON configuration file, documented here:

  • Spotfire Streaming > Configuration Guide > Runtime Configuration Types > Runtime Application Configuration

Correct this by increasing the Transactional Memory that a node is allowed to use by using two configuration files, the ApplicationDefinition and NodeDeploy HOCON configuration files.

In the ApplicationDefinition, increase the "memorySizeBytes" value to big enough to support the "Max Object Size" needed. In the NodeDeploy configuration use the ApplicationDefinition through the nodeTypes setting.

ApplicationDefinition .conf example:

name = "my_application"
version = "1.0.0"
type = "com.tibco.ep.dtm.configuration.application"
configuration =
{
  ApplicationDefinition = {
    execution = {
      nodeTypes = {
        "medium-capacity" = {               
          sharedMemory = {
            memoryAllocators = 16
            memoryType = FILE
            memorySizeBytes = 4g
          }
        }                            
      }
    }
  }
}

NodeDeploy configuration example:
name = "NodeDeployment"
version = "1.0"
type = "com.tibco.ep.dtm.configuration.node"
configuration = {
  NodeDeploy = {
    nodes = {
      "${EP_NODE_NAME}" = {
        nodeType = "medium-capacity"
      }
    }
  }
}
 

This example has given the nodeType configuration the arbitrary name "medium-capacity". You may name this anything with typical alphanumeric characters.

Use "${EP_NODE_NAME}" for the node name in the NodeDeploy configuration instead of a specific "{nodename}.{groupname}.{clustername}" value (such as "A.USWEST.BIZAPP") to match any node and cluster you provide on the 'epadmin install node' command.

The ApplicationDefinition .conf file goes into the project "src/main/configurations" folder. The NodeDeploy .conf may also be in the project at the same place, or it may be maintained outside the project and referenced on the command-line as so:
epadmin install node --nodename=A.X --application=app.zip --nodedeploy=myNodeDeploy.conf

For more about Transactional Memory, please see the Spotfire Streaming product documentation:

  • Spotfire Streaming > Concepts Guide > Transactional Memory
  • Spotfire Streaming > Authoring Guide > Using Query Tables > Persistence of Query Tables, "Transactional Memory Option"
  • Spotfire Streaming > Architecture Guide > Transactions

Issue/Introduction

Configuration guidance