book
Article ID: KB0074801
calendar_today
Updated On:
Description
We started twelve TIBCO Streaming nodes on the same Linux system and could not start another one. Additional nodes failed at startup with error:
java.lang.OutOfMemoryError: unable to create new native thread
Issue/Introduction
Configuration and troubleshooting guidance
Resolution
This error is caused by reaching the operating system user limit for the number of concurrent threads allowed in a single session.
The limit is reported by running 'ulimit -u' and can be increased by an administrator running command:
ulimit -SHu unlimited
The 'S' stands for the user soft-limit. The 'H' stands for the operating system hard-limit which the soft-limit cannot exceed. The 'u' stands for the "max user processes".
We observed that on Red Hat 8, the upper limit was reached even though 'ulimit -u' reported 'unlimited'. In this case the solution is to start a portion of the nodes as a different user. The nodes will still discover each other, join their cluster, and communicate over the network normally. Each user will need to manager their own set of nodes when the nodes require authentication.
Other causes of this error are related to running out of available physical RAM because the JVM Heap setting (-Xmx) of all the JVM's (one or more per node) take too much memory and there is too little left for Native I/O (disk and network), off-heap memory (allocated from native libraries), and process call-stacks. In these cases, reduce the sum of all JVM Heap memory settings (as allowed by the application). Reducing the Heap too much may cause slow Streaming compilation, add tuple processing latency, and result in running out of Heap space.
Another solution is to reduce the number of separate processes within a node by reducing concurrency and multiplicity. A final advanced solution is to reduce the JVM thread stack-size (-Xss setting). If using either of these techniques, careful monitoring of the application under business loads is required to ensure that these changes do not result in incorrect application logic, unacceptable performance, or crashes due to being out of stack space.