Resolution:
In FabricServer 2.5 with WebSphere 6.1, Node agent crash detection triggers shutdown when node agent is still running.

We have noted the following symptoms a few times on Windows Engines:

07/26/07 19:55:35.625 Severe: While checking if Node Agent is running: java.net.BindException: Address already in use: connect
07/26/07 19:55:35.625 Severe: An unexpected crash detected for the Node Agent… will notify listeners and stop monitoring.

07/26/07 19:55:35.625 Info: Deactivating domain.
07/26/07 19:55:35.625 Info: Will deactivate "WAS_Generic_mat_90" domain.

But the nodeagent is still running.

See logs in \\wopr\testing\Nube\fabric\

In broker log:
1) 7/26/07 16:33:01.289 Warning: The container hosting domain "WAS_Generic_mat_90" on engine "qapc55-1" has crashed: "Node Agent crashed"
qapc55-1 detects nodeagent crash due to BindException: Address already in use: connect but Nodeagent continues to run under qapc55-3

2) 07/26/07 19:57:10.262 Warning: The container hosting domain "WAS_Generic_mat_90" on engine "qapc41-1" has crashed: "Node Agent crashed"
qapc41-1 detects nodeagent crash due to BindException: Address already in use: connect and is the instance running the node agent. When it exists the node agent process dies and all other instances detect node agent crashes and none of them are able to succeed restarting (as configuration hasnt been cleaned up on DM)

Using Container 2.0.0.7
Distro 2.0.0.5_2.0.0.3

Possible Solution

There are a fixed number of outbound ports that can be used for establishing TCP/IP connections, and they are only available for re-use after some timeout period (typically a few minutes) after they've been closed. On some flavors of Microsoft Windows (including Windows 2003 Server edition), the default maximum allowed number of such ports is just 5000. So, update the "MaxUserPort" TCP/IP parameter in Windows Operating System registry entry to a higher value (may be the highest allowed 65534) to allow more than 5000 ports to be open at a time.

Also, look at this support document from Microsoft Corp. for updating the relevant registry entry:
http://support.microsoft.com/default.aspx?scid=kb;en-us;196271