Resolution:
Description:
============

BwEngine process started holding maximum LWPs on secondary server.


Environment:
===========

Solaris 10

Symptoms:
========

Listing down the process on the box by using prstat will show many BW processes LWPs.

Sample :

PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND

  9723 tibadmin 999  47    4 1377M 1206M sleep   34:21  0.00% bwengine

  6214 tibadmin 999  59    0  780M  609M sleep   48:13  0.00% bwengine

  9727 tibadmin 999  47    4  566M  415M sleep   26:18  0.00% bwengine

  8793 tibadmin 999  59    0  591M  402M sleep   17:25  0.00% bwengine

10200 tibadmin 999  59    0  533M  381M sleep   17:24  0.00% bwengine

11513 tibadmin 999  59    0  474M  331M sleep   26:49  0.00% bwengine

14240 tibadmin 999  59    0  498M  316M sleep   24:32  0.01% bwengine

  9567 tibadmin 999  59    0  482M  311M sleep   15:50  0.00% bwengine

26780 tibadmin 185  59    0  447M  308M sleep   30.6H  0.01% hawkagent_ISBPr

14248 tibadmin 999  59    0  499M  305M sleep   15:55  0.00% bwengine

  9761 tibadmin 121  59    0  487M  293M sleep   85:24  0.01% bwengine

20583 tibadmin 135  59    0  548M  292M sleep   30:12  0.00% bwengine

  2685 tibadmin 126  59    0  425M  289M sleep    2:08  0.00% bwengine

  9563 tibadmin 999  59    0  459M  283M sleep   15:43  0.00% bwengine

15749 tibadmin  99  59    0  470M  280M sleep   25:53  0.00% bwengine


Cause:
=====

When the secondary bwengine comes up due to a loss of a heartbeat from the primary, it creates LWPs. These LWPs are left out when such failover happens more frequently due to networks delays.


Resolution:
==========

Tune heartbeatInterval, activationInterval and preparationInterval as follows. This will delay the secondary instance to come up due to network delays.

Hearbeat = X
Preparation Interval = 2x
Activation interval = 3x

BW processes are holding maximum threads (LWP) on secondary BW server.