Problem Reported :Unable to login to CIS Studio to one of production node, the screen gets freezes after sometime. Also manager URL is not accessible via browser although all services are up and running when we check in server. 9400 port is open and listening to requests.

Also we tested few odbc request with the host which it unable to serve. 

Business case : 
With this node unable to serve user request this will increase load on other active node and can causing extra overhead for the platform. 

Error:

ERROR [DbChannel-RequestSelectThread] 2017-11-29 06:00:36.930 -0500 RequestSelectThread - Failed to accept client connection.[dbchannel-1918039] [dbchannel-1918039] at com.compositesw.server.dbchannel.RequestSelectThread.run(RequestSelectThread.java:415) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at com.compositesw.server.dbchannel.RequestSelectThread.run(RequestSelectThread.java:380) ... 3 more

In this environment we have a 2 nodes in cluster.

The issue was observed on both of nodes and both were not responding although services were up.

We restarted services on one of the node and the server was able to respond normally after that. Did not restarted the 2nd node to find the root cause. When I access the manager page and check under cluster management, I see the status showing for the active node is BLOCKED and for the second node it is in BLOCKING status.

Solution : Had a troubleshooting session and we have increased the ulimit size. Then restarted the 2nd node and issue got resolved. Customer has observed the cluster and platform for week and issue did not re-occur and closed the case.