Resolution:
Description:
============
BW Engine throws, “unable to create new native thread”, even after reducing the stack size to a lower value.

Environment:
===========
BW 5.8, SunOS 5.10 Generic_147440-25 sun4v sparc


Symptoms:
========
Reducing the stack size value to 128k for a number of bwengines (the system is running around 100 bwengines at once) did not resolve the issue.

A related error was noticed in the Solaris command line shell - the system is sometimes unable to fork a new process. After investigation, it appears that Solaris was running out of kernel pageable memory.

kstat -n segkp

module: vmem                            instance: 170
name:   segkp                           class:    vmem
        alloc                           155041347
        contains                        0
        contains_search                 0
        crtime                          5820046.68360304
        fail                            127112
        free                            155010659
        lookup                          62712240
        mem_import                      0
        mem_inuse                       2108432384
        mem_total                       2147483648
        populate_fail                   0
        populate_wait                   0
        search                          225041364870
        snaptime                        5872950.73221698
        vmem_source                     0
        wait                            331674334

The above statistics from the OS shows that the process is exceeding 2GB (OS default) of kernel pageable memory. The number of Lwps were all seen to be high, at approximately 17000.


Cause:
=====

The amount of LWPs depends on the amount of RAM available, more precisely, the size of the segkp (pageble memory). The issue was caused by the default pageble memory size of 2G.


Resolution:
==========

Increase the sigkp(pageble memory) to 8G.

java.lang.OutOfMemoryError: unable to create new native thread on Solaris.