Dequeuing client's memory allocation, has exceeded the maximum
Article ID: KB0076117
Updated On:
| Products | Versions |
|---|---|
| TIBCO Streaming | - |
Description
A network client is disconnected by the server with the error messages:
Queue timeout: deadlock detection timed out after 1,000 milliseconds, assumed to be an external deadlock, queue expanding by 32,768 tuples (approximately NNN bytes) StreamBase.SB.StreamBaseException: Dequeuing client's memory allocation of: 8.12 mbytes, has exceeded the maximum memory allocation of: 8 mbytes. This may be a sign that the client is hung.
Issue/Introduction
Dequeuing client's memory allocation, has exceeded the maximum
Resolution
REFERENCE: page-pool (http://docs.streambase.com/latest/index.jsp?topic\=/com.streambase.sb.ide.help/data/html/reference/sbdxml.html%23sbdxml_pagepool)
These messages indicate that a dequeuing client is slower than the server, and the space allocated to store outgoing tuples from the server have filled all of the allocated space.
The client is any application that has subscribed to a stream over the network. The client may be a StreamBase Client API application or another StreamBase Server instance using a remote container connection.
Every dequeue client connection has a queue maintained by the sbd service which is drawn from the JVM heap (-Xms -Xmx settings). The server limits the amount of output queue memory allowed for each client so a misbehaving client that subscribes but never (or rarely) dequeues will not force the server to exhaust all of its memory. This limit protects the server.
The maximum amount of queue memory allowed each client is set in the <page-pool> node of the sbd.sbconf configuration file:
In a correctly functioning ideal system where the consuming clients can process more messages each second than the emitting server, the output queue will stay a low number of pages.
In most systems, occasionally for brief periods the client is slower than the server, either due to a temporary lack of a resource on the client, or a burst of data from the server. In this case, messages will queue in the space provided. The expectation is that the situation is brief enough that the client will be able to catch up to the server and start drawing down the queue to minimum size again.
If the server is constantly faster than the client, increasing the queue size only delays the inevitable disconnection.
If disconnected, the client application should call StreamBaseClient.close() and attempt to reconnect. It will miss the messages that were queued since those are dropped at disconnection. The client will likely get into trouble again if higher data-rates than it can consume are sustained.
Note that external clients using the Java, C++, or C# StreamBase Client API's create an internal thread that dequeues automatically from the server up to 10000 (by default) messages. It is the responsibility of the programmer to call StreamBaseClient.Dequeue() with sufficient frequency to consume from this internal queue. Only when this queue is full will queuing in an output queue begin on the StreamBase server for that client.
To keep a client from being disconnected you may:
a) Let the server queue a larger amount of data before disconnecting a client by increasing the server max-client-pages setting. Do not change the default page-size setting.
b) Have the server throttle the output to a rate the client can consume so the server does not queue a lot of data at the output stream. To throttle the output, you may store the tuples in an In-Memory Query Table (local to the StreamBase application), and then query that table to send results in smaller/less-frequent batches to your clients. The 'Data Cache Framework', available on the StreamBase Component Exchange, provides an example of this procedure. To load this component into your Studio workspace, go to File > StreamBase Component Exchange > (search) Data Cache Framework.
c) Increase the speed of the client to consume more messages each second from the server. (Note: If the network capacity is full causing the server to queue output, then changes to the client will not help.).
When tuning the application you may monitor connection health by repeatedly running sbadmin listConnections. Look at the 'memory=' value to see which clients are causing the server to queue.
On Windows:
On Linux:
The output of sbadmin listConnections looks like:
These messages indicate that a dequeuing client is slower than the server, and the space allocated to store outgoing tuples from the server have filled all of the allocated space.
The client is any application that has subscribed to a stream over the network. The client may be a StreamBase Client API application or another StreamBase Server instance using a remote container connection.
Every dequeue client connection has a queue maintained by the sbd service which is drawn from the JVM heap (-Xms -Xmx settings). The server limits the amount of output queue memory allowed for each client so a misbehaving client that subscribes but never (or rarely) dequeues will not force the server to exhaust all of its memory. This limit protects the server.
The maximum amount of queue memory allowed each client is set in the <page-pool> node of the sbd.sbconf configuration file:
<page-pool>
<param name="page-size" value="4096"/>
<param name="max-client-pages" value="2048"/>
</page-pool>
<param name="page-size" value="4096"/>
<param name="max-client-pages" value="2048"/>
</page-pool>
In a correctly functioning ideal system where the consuming clients can process more messages each second than the emitting server, the output queue will stay a low number of pages.
In most systems, occasionally for brief periods the client is slower than the server, either due to a temporary lack of a resource on the client, or a burst of data from the server. In this case, messages will queue in the space provided. The expectation is that the situation is brief enough that the client will be able to catch up to the server and start drawing down the queue to minimum size again.
If the server is constantly faster than the client, increasing the queue size only delays the inevitable disconnection.
If disconnected, the client application should call StreamBaseClient.close() and attempt to reconnect. It will miss the messages that were queued since those are dropped at disconnection. The client will likely get into trouble again if higher data-rates than it can consume are sustained.
Note that external clients using the Java, C++, or C# StreamBase Client API's create an internal thread that dequeues automatically from the server up to 10000 (by default) messages. It is the responsibility of the programmer to call StreamBaseClient.Dequeue() with sufficient frequency to consume from this internal queue. Only when this queue is full will queuing in an output queue begin on the StreamBase server for that client.
To keep a client from being disconnected you may:
a) Let the server queue a larger amount of data before disconnecting a client by increasing the server max-client-pages setting. Do not change the default page-size setting.
b) Have the server throttle the output to a rate the client can consume so the server does not queue a lot of data at the output stream. To throttle the output, you may store the tuples in an In-Memory Query Table (local to the StreamBase application), and then query that table to send results in smaller/less-frequent batches to your clients. The 'Data Cache Framework', available on the StreamBase Component Exchange, provides an example of this procedure. To load this component into your Studio workspace, go to File > StreamBase Component Exchange > (search) Data Cache Framework.
c) Increase the speed of the client to consume more messages each second from the server. (Note: If the network capacity is full causing the server to queue output, then changes to the client will not help.).
When tuning the application you may monitor connection health by repeatedly running sbadmin listConnections. Look at the 'memory=' value to see which clients are causing the server to queue.
On Windows:
@ECHO OFF
:loop
echo "---"
sbadmin listConnections
timeout /t 10
goto loop
:loop
echo "---"
sbadmin listConnections
timeout /t 10
goto loop
On Linux:
while (true); do echo "---"; sbadmin listConnections; sleep 2; done
watch -n 10 'sbadmin listConnections'
The output of sbadmin listConnections looks like:
id=1BABB9ADB306C6DFF512E57620BB3018; peer=/127.0.0.1:42762; local=/127.0.0.1:10000; memory=0bytes; subscriptions=default.BestAsk,default.BestBids,default.Quotes,system._internal_control,system.error; dequeued=3; enqueued=0; dequeuePackets=2; enqueuePackets=0; protocolFamily=7.2; status=OK/OK
field | contents |
|---|---|
peer | IP address and port of the client |
local | IP address and port of the server |
memory | the size of the output queue for this client |
subscriptions | the streams to which this client has subscribed |
dequeued | the number of messages this client has received |
enqueued | the number of messages this client has sent |
dequeuePackets | the number of packets (which contain 0 or more tuples) this client has received |
enqueuePackets | the number of packets this client has received |
protocolFamily | the version of the StreamBase wire-protocol used |
status | the health of the client connection |
Feedback
Yes
No
Powered by
