A spike is seen when running a command from the EMS Admin tool, when there is a performance issue with EMS caused by either a slow disk or misbehaved clients.
Article ID: KB0085444
Updated On:
| Products | Versions |
|---|---|
| TIBCO Enterprise Message Service | - |
| Not Applicable | - |
Description
Resolution:
EMS use the OS provided I/O event notification facility for networking. The characteristics of this facility are that on each notification there can be multiple file descriptors
The request of a "show server" command at this point will be transmitted to the EMS server TCP buffer, but it will not get processed until EMS gets out of the current event handling process. Events can be accumulated if the previous event handling takes too long. This will further increase the servicing time, especially when there are a large number of connections to the EMS server.
A misbehaved client impacts the service time of EMS to other clients in the same way as a slow disk does. Such a client includes but is not limited to clients that are doing the following:
- A client use FT connection, and it creates a massive number of sessions | producers | consumers
- A client with FT connection who creates and destroys sessions | producers | consumers repeatedly
- A client who sends large messages.
- An Admin client who runs the "show consumer" | "show queues" ... commands ( or the equivalent Admin API ), and the number of consumers | queues on EMS is large.
EMS use the OS provided I/O event notification facility for networking. The characteristics of this facility are that on each notification there can be multiple file descriptors
that are ready for recv(). On Linux for example, each time epoll_wait() returns, EMS processes all the fds that are ready in a loop. It then proceeds to call the next epoll_wait().
The possible effect of a slow write():
Assume at a given point epoll_wait() returns with 10 fds ready for reading because 10 producers are publishing messages. EMS will then read the data from the TCP buffer and do the regular deserialization, then process internally and finally write to the datastore. A normal write() completes in 200 microseconds but now takes 100 milliseconds. Because of that, for this batch of notifications, EMS takes at least 1 second to return.
The possible effect of a slow write():
Assume at a given point epoll_wait() returns with 10 fds ready for reading because 10 producers are publishing messages. EMS will then read the data from the TCP buffer and do the regular deserialization, then process internally and finally write to the datastore. A normal write() completes in 200 microseconds but now takes 100 milliseconds. Because of that, for this batch of notifications, EMS takes at least 1 second to return.
The request of a "show server" command at this point will be transmitted to the EMS server TCP buffer, but it will not get processed until EMS gets out of the current event handling process. Events can be accumulated if the previous event handling takes too long. This will further increase the servicing time, especially when there are a large number of connections to the EMS server.
A misbehaved client impacts the service time of EMS to other clients in the same way as a slow disk does. Such a client includes but is not limited to clients that are doing the following:
- A client use FT connection, and it creates a massive number of sessions | producers | consumers
- A client with FT connection who creates and destroys sessions | producers | consumers repeatedly
- A client who sends large messages.
- An Admin client who runs the "show consumer" | "show queues" ... commands ( or the equivalent Admin API ), and the number of consumers | queues on EMS is large.
Issue/Introduction
A spike is seen when running a command from the EMS Admin tool, when
there is a performance issue with EMS caused by either a slow disk or
misbehaved clients.
Feedback
Yes
No
Powered by
