Inference Agent does not process messages from the Solace queue.Error Response (403).

Inference Agent does not process messages from the Solace queue.Error Response (403).

book

Article ID: KB0093086

calendar_today

Updated On:

Products Versions
TIBCO BusinessEvents Enterprise Edition -
Not Applicable -

Description

Description:
InferenceAgent does not process messages on the JMS queue (Solace) - restart of InferenceAgent does not help. The following error information is logged.


Error Response (403) - Client Name Already In Use and Log file includes warnings "Thread pool [<channelURL>] is overloaded and the job submissions are facing repeated failures."


Symptoms:
Inference Agent does not process messages from the Solace queue.

When initializing the JMS channel (startup), the following information is logged.


<timestamp> <agent> Info [main] - [com.solacesystems.jcsmp.protocol.impl.ChannelOpStrategyClient] Error Response (403) - Client Name Already In Use
<timestamp> <agent> Info [main] - [driver.jms] Error setting client id - client id already in use (403: Client Name Already In Use): ClientID = <clientID>. Setting new ClientID =<clientID>-1
<timestamp> <agent> Info [main] - [com.solacesystems.jms.SolConnection] Entering setClientID(), clientId: <clientID>-1_Queue


Cause:
Some old (zombie) processes (InferenceAgents)  were still running on the machine. Those processes have not been marked as zombie processes in ps output. Those processes are still connected/subscribed to the Solace queues with the same client ID as the new started InferenceAgents.  On Solace, the new Subscriber (same client ID) is unable to load  messages from JMS queues.

Resolution

Kill the zombie processes. To identify them:
 
execute (be-engine -> not recognized by Administrator):
ps -ef|grep be-engine|grep <application name>

or

ps commands to find zombie processes:
ps -ef | grep defunct
ps aux | awk '$8=="Z" { print $2 }'

Issue/Introduction

Inference Agent does not process messages from the Solace queue.Error Response (403).