If bwagent (started as a Linux service) on a given server crashes or is intentionally killed with kill -9 all the AppNodes running on this server stop immediately and with no message in their respective logs.

 

If bwagent on a given server crashes or is intentionally killed with kill -9 all the AppNodes running on this server stop immediately and with no message in their respective logs.

Linux

The problem happens if the 'bwagent' is started as a Linux service and not properly configured. By default Linux services are killed with the 'control-group' option and all child processes are killed when the service is stopped. This can be changed by using KillMode=process option in the service block of the service file.

In "/etc/systemd/system" folder edit the 'service' file to add the "KillMode=process" option:

[Service]
Type=simple
User=tibco
WorkingDirectory=/opt/tibco/bw/6.4/bin/
ExecStart=/opt/tibco/bw/6.4/bin/bwagent
Restart=on-failure
KillMode=process

Then make sure the change is taken into account with the following command:
systemctl daemon-reload

The expected behavior is that once started the AppNodes are running as independent OS level processes and will not be impacted by what happens to the bwagent.

https://stackoverflow.com/questions/32208782/cant-detach-child-process-when-main-process-is-started-from-systemd 
https://serverfault.com/questions/700862/do-systemd-unit-files-have-to-be-reloaded-when-modified 
https://www.freedesktop.org/software/systemd/man/systemd.kill.html