When there are pending messages in the EMS server, while starting consumers with a selector, high CPU usage and high disk read rate with EMS 5.1.0+ was noticed. The same symptoms were not seen with pre EMS 5.1.0 versions.
Article ID: KB0087942
Updated On:
| Products | Versions |
|---|---|
| TIBCO Enterprise Message Service | - |
| Not Applicable | - |
Description
Resolution:
When a queue consumer with a selector was started and there were pending messages in the queue, all messages were reevaluated against the selector. This sometimes led to swapping in messages from disk. If a message did not pass the selector evaluation, and therefore was not sent to the consumer, it is now swapped back to disk immediately which reduces the overall message memory usage. This also fixes a performance-related issue where a queue with consumers that had selectors running at different speeds (or a mix of consumers with and without selectors) could reach a level of message memory usage that caused the server to swap messages back to disk. This should be the expected behavior with 5EMS .1+.
The change request was created because users complained that message memory usage was high. The fix was to swap a message out after selector evaluation if the message could not be sent. If another consumer starts with a selector, those messages need to be evaluated. Before the change request the messages may or may not still be in memory so there was no need to read. After the change request, the messages most likely will not be in memory so a read is required.
Frequently creating consumers with a selector on a queue with pending messages could lead to poor performance. The old behavior may have been acceptable with EMS 4.3 when there were not many pending messages and the other destinations did not have much data. This was because it wa likely that the server would not have to swap messages out once they were evaluated for the first time. However, that still could happen when "Message Memory Usage" is high, that is, even with EMS 4.3, if messages accumulated, you could see an issue of server lock-up while processing the consumer start up.
TIBCO has planned for a parameter that would disable the new behavior. This parameter is not documented. Use the following to enable the old behavior:
“swapout_non_sent_queue_msgs = false” in the main config file (tibemsd.conf)
Run a test in your environment to see if this will address the stated issue. Be aware that creating/destroying queue consumers with selectors frequently can cause degradation in performance if the queue backlog grows large. This is true with or without the flag.
When a queue consumer with a selector was started and there were pending messages in the queue, all messages were reevaluated against the selector. This sometimes led to swapping in messages from disk. If a message did not pass the selector evaluation, and therefore was not sent to the consumer, it is now swapped back to disk immediately which reduces the overall message memory usage. This also fixes a performance-related issue where a queue with consumers that had selectors running at different speeds (or a mix of consumers with and without selectors) could reach a level of message memory usage that caused the server to swap messages back to disk. This should be the expected behavior with 5EMS .1+.
The change request was created because users complained that message memory usage was high. The fix was to swap a message out after selector evaluation if the message could not be sent. If another consumer starts with a selector, those messages need to be evaluated. Before the change request the messages may or may not still be in memory so there was no need to read. After the change request, the messages most likely will not be in memory so a read is required.
Frequently creating consumers with a selector on a queue with pending messages could lead to poor performance. The old behavior may have been acceptable with EMS 4.3 when there were not many pending messages and the other destinations did not have much data. This was because it wa likely that the server would not have to swap messages out once they were evaluated for the first time. However, that still could happen when "Message Memory Usage" is high, that is, even with EMS 4.3, if messages accumulated, you could see an issue of server lock-up while processing the consumer start up.
TIBCO has planned for a parameter that would disable the new behavior. This parameter is not documented. Use the following to enable the old behavior:
“swapout_non_sent_queue_msgs = false” in the main config file (tibemsd.conf)
Run a test in your environment to see if this will address the stated issue. Be aware that creating/destroying queue consumers with selectors frequently can cause degradation in performance if the queue backlog grows large. This is true with or without the flag.
Issue/Introduction
When there are pending messages in the EMS server, while starting consumers with a selector, high CPU usage and high disk read rate with EMS 5.1.0+ was noticed. The same symptoms were not seen with pre EMS 5.1.0 versions.
Feedback
Yes
No
Powered by
