Resolution:
Description:
============
Sometimes you may notice a gradual slowdown of the message consumption rate when a destination reaches a large queue depth (along with large destination storage). This impacts on message production and consumption ratio and can slow down EMS operations.

Symptoms:
========
In the output of ‘info’ or ‘show server’ command, if you find the inbound rate is greater than the outbound rate then that is an indication that there are slow consumers (pending messages and size will also be high). There will be an overall slow rate of message consumption witnessed in the EMS server.

Resolution:
==========
1). Find the slow consumer.

You can use the results from the commands ‘show stat consumers’ and ‘show connections  user=&ltuser name>’  to find the slow consumers. Use the connection ID from the ‘show connections’ command and look for the same connection id from ‘show stat consumers’ command. Compare the message rate with other connections from other hosts to find out who are the slower consumers causing the pending messages.

2). Confirm if the above consumer is really a slow consumer.

To confirm that a consumer is slow at processing messages off a queue or topic, run the following command on the EMS admin console:

‘show consumers full queue=&ltqueue name>’ or ‘show consumers full topic=&lttopic name>’

Compare the Last Acked time (Last Akcd  is the approximate time elapsed since last time any message sent to the consumer was acknowledged by consumer's session. Value is approximate with precision of 1s) with the last Sent time (the last time the EMS broker sent a message to the consumer).

If the Last Acked time is significantly longer than the Last Sent time, then your consumer is slow.

Example:.

Last Acked = 3000s (i.e. 50 minutes ago)
last Sent = 2s

In order to improve the performance of slow consumers you should check the following:

a). Is the client using client-side message selectors?

From the output of the 'show consumers full' command above, check the SASB column. If you see a + in the 3rd column, the client is using a selector.

Example:

SASB
+A+-

Client-side message selectors can degrade EMS performance.

b).Is the client using Client acknowledgement mode?

From the output of the 'show consumers full' command above, check the SASB column. If you see a C in the 2nd column, the client is using CLIENT_ACKNOWLEDGMENT.

Example:

SASB
+C--

Client acknowledgement mode is blocking so the next request is not processed until the current request is  confirmed. Check the application to confirm why the current request is not being processed quickly.

cI). Check system resources on the client host. Capture and monitor the CPU, Memory and networking statistics and check them for issues.

d). If you still have not find the root reason of the problem, you should check what the application is doing, e.g. if the application needs to access database and database issue causes slow consumer issue.

Environment:
===========
All

Detecting and confirming slow consumer clients using EMS Admin commands.