Input messages for specific destination(s) not processed anymore or processed very slowly by TIBCO BusinessEvents Inference Agents .
Warning(s) "Thread pool [<channelURL>] is overloaded" reported in log file constantly at specific interval of 120 seconds. 

Handling of Warning "Thread pool [] is overloaded" reported in TIBCO BusinessEvents log file

All Operating Systems

These issues can be caused by a deadlock situation (resource not available)  or by your project code (ex. locking -> Cluster.DataGrid.Lock()).
To be able to find the root cause of the issue:
 
1.)
Create some thread dumps for the inference agent (5 dumps, interval of 30secs). You can use JDK tool "jstack <pid>" to create the thread dumps.
Validate worker threads handling the input destination in dumps to find out why/where threads are locked. You can also collect thread dumps from JMX tool JVisual VM after connecting to the inference agent. Please note the agent should have JMX enabled to connect to it from JMX tool like JVisual VM.
 
2.)
When that will not help. You can increase the log level.
 
There are three options:
2.1.)
In CDD (restart of InferenceAgent required)
 
CDD -> Collection TAB -> LogConfiguration -> <logConfigOfInferenceAgent> -> Set Level from "*:info" to "*:debug"
 
NOTE:
Increase the log file size too
 
2.2.)
When JMX is enabled for the InferenceAgent (CDD property: be.engine.jmx.connector.port) connect to the agent using Java JDK tool jVisualVM or JConsole and execute below operation.
 
Enable debug:
MBeans -> com.tibco.be -> Methods -> Engine -> Operation SetLogLevel, parameters: * , debug
 
Disable debug:
MBeans -> com.tibco.be -> Methods -> Engine -> Operation SetLogLevel, parameters: * , info
 
3.) HAWK method SetLogLevel("*", "<level>")
<level> ... debug or info
 
Note:
Some common scenarios when we see this warning in logs are database becoming slow due to stuck transaction on DB end, Cache responding slowly to requests, locking issues caused by bad design or stuck/long running operations. If you could not identify the specific case then please open a support case to assist with thread dump analysis. Please attach your project, the CDD file used to start the agent, the log file and the thread dumps to the support case.
 

RTC, ThreadPool