Resolution:
By default, RTserver sets the option Client_Max_Buffer to 10MB. The Client_Max_Buffer setting is the upper bound on the size of the write buffer in RTserver associated with each RTclient connection. If you have fifty RTclients connected to the RTserver then there will be fifty write buffers allocated in RTserver, one for each connection.

The RTserver puts messages it receives from RTclient publishers in these write buffers before pushing them out to the subscribing RTclients. If there is no room to push messages out to an RTclient (meaning that both the RTclient's 48k socket buffer and RTserver's 48k socket buffer are full) then the messages remain in this write buffer until the RTclient consumes the messages queued up in the respective socket buffers, which frees space for the RTserver to push more messages to the RTclient.

Initially the write buffer size for each connection in RTserver is 8K but if messages are sent faster by the publishing RTclient than the subscribing RTclient can consume them then this buffer in RTserver grows, allocating additional buffer space up to whatever the client_max_buffer option is set to (10Mb by default). At some point, if the subscribing client can consume messages faster than message are being published to it, then the write buffer associated with the connection in RTserver will get drained and RTserver will reclaim the memory allocated for the buffer.  

Recommendations:
1) Stop RTserver, increase client_max_buffer by 25%, restart RTserver: This is the least intrusive approach, the idea being that maybe RTclient needs just a little more space on the RTserver to balance off its slow message processing against a fast Publisher.
2) Throttle back your Publishing RTclient: The Publishing RTclient is sending messages faster than the Subscribing RTclient can process them. Put something like a TutSleep(.25) just before TipcSrvMsgSend() to slow down the Publishing RTclient.
3) Speed up your Receiving RTclient: Use Process Callbacks instead of Subject Callbacks - they're faster. Limit the work the callbacks do in the receiving RTclient. The idea being that a Receiving RTclient should be dedicated to pulling messages off of the RTserver and then moving them to either shared memory or file where some other process would do the actual work with the messages.
4) Introduce Load Balancing: Clone your subscribing RTclient and divide the work among two or more identical subscribing RTclients.
5) A combination of any or all of the above.

Overrunning an RTclient