Resolution:
This is the expected behavior. TIBCO EMS Server only accounts for the actual memory used for holding message information, not for memory used for other purposes, such as lists, hash tables, the index of written records, etc.  This can be substantial as the sheer number of messages grows.

The EMS Server process will be always substantially bigger than the value set by max_msg_memory. Still using that parameter will effectively limit the process size. Indeed, while the server process size will be bigger, it will be limited at *some* point and will not grow beyond. How much bigger is difficult to say and it is heavily platform-dependent.

If you want to use max_msg_memory to limit the actual tibemsd process size, you should measure the actual tibemsd size on a given platform, for a given max_msg_memory, for a given deployment scenario (nature of the messages, size, delivery mode, etc). You should test your particular scenario on your deployment platform and check what the actual size will be for a given max_msg_memory value. Once tested, the tibemsd size will be always pretty close to that value.