Resolution:
Check Your Thread Stack Size :
=========================
The first thing you might try in the checklist is reducing the stack size. The JVM has an interesting implementation, by which the more memory is allocated for the heap (not necessarily used by the heap), the less memory available in the stack, and since threads are made from the stack, in practice this means more “memory” in the heap sense (which is usually what people talk about) results in less threads being able to run concurrently.
To reduce the stack size, add “-Xss64kb” to the JVM options. I suggest you start with 64k, try the application, then if it doesn’t work (it will fail with a Java.lang.StackOverFlowError), increase the stack to 128k, then 256k, and so on. The default stack size is 8192k so there’s a wide range to test.
Checking your OS limits.conf :
========================
At operating system level, Linux users you can control the amount of resources (and in particular file descriptors) you are going to use in the limits.conf file or usint the ulimit command:
# vi /etc/security/limits.conf
testuser soft nofile 4096
testuser hard nofile 10240
# ulimit -Hn
10240
In either case, if the amount of file descriptors you are allowed to use is less the number of threads you are going to create, then this could be the issue.
Checking the number of Threads created :
=================================
How do you check the number of threads created by your application ? You can do it in at least two ways:
$ ps -p PID -lfT | wc -l
The above shell will return the number of Lightweight Processes created for a Process indicated by the PID. This should match with the Thread Dump count generated by jstack:
$ jstack -l PID | grep tid | wc -l
Once that you have counted the number of threads, then you should verify that you are not hitting system limits, specified by the kernel.pid_max limit parameter. You can check this value by executing:
$ sysctl -a | grep kernel.pid_max
kernel.pid_max = 32768
Finally, you have also to check if your OS allows you enough processes for user. This can be checked through:
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515005
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
The default number of process per users is 1024 by default. So adding the following to your $HOME/.profile could solve the issue:
ulimit -u 50000 (or more according to your load )