Resolution:
The max_heap_size is constrained by the Operating System and the RAM size and normally to determine the maximum heap size setting for server instances, you use the following equation viz;

-> TOTAL_RAM / 2 / NUMBER_OF_SERVERS = MAX_HEAP_SIZE

(e.g. 8GB RAM on a single server instance can theoretically set MAX_HEAP to 4GB i.e. 8/2/1=4 ).

The max_heap_size is further limited by the the type of JVM being run whether its 32bits or 64bits with 64bits generally allowing a bigger size to be set.

Setting the heap size too high could have adverse effect on the operation of the Broker as the garbage collection "Pauses" tends to be longer and this will require the correct configuration to accommodate connected Drivers and Engines.

Also in troubleshooting and when considering whether to increase max_heap_size to accomodate or ameliorate memory issues, one must first checked if some Services have the setting of deleteInvocationData set to SERVICE_REMOVED. This is a common reason why garbage collection dont reclaim enough memory from the heap as the invocation data is stored in memory until Service Instance is removed either manually from the Admin Console or automatically according to the Service Cleanup setting. The default option for deleteInvocationData of SERVICE_COMPLETED is recommended and advisable given this scenario.

Ref:

http://docs.sun.com/app/docs/doc/819-4742/6n6sfgml7?l=en&a=view#abeij

http://www.jroller.com/mert/entry/java_heapconfig_32_bit_vs

http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx

Guidelines for Max Heap Size setting and Memory Issues Troubleshooting