1). BC platforms support 64-bit JREs, so you can increase the max heap size above 2048 MB. Try setting this value below the maximum memory size of the machine.  Also, make sure the thread stack size is set to no more than 256 KB and the number of threads are set to at least 40.

2). By default, BC sets the engine for no "throttling" which can lead to thrashing and excessive memory usage within the BC engines themselves, especially if the transaction completion rate exceeds the transaction input rate. Try setting throttling for both inbound and outbound transactions.

For inbound, uncomment the following line in all deployed engines and set it to a finite value:

MaxJobs.Gateway/MSH/Inbound/From-DMZ/JMS\ Receiver.process=10 (for BusinessConnect 7.X installations)
 

MaxJobs.Gateway/MSH/Inbound/From-DMZ/RV\ Receiver.process=10  (for BusinessConnect 6.X installations)

This will limit the active number of inbound jobs that can be run by the interior server at any given time. You can experiment with this value to see what works best.

Similarly, you can limit the number outbound requests that can be processed by modifying this parameter:

MaxJobs.Gateway/BMH/Outbound/From-Back-Office/JMS/Request\ Receiver.process=10 (for EMS usage)

MaxJobs.Gateway/BMH/Outbound/From-Back-Office/RV/Receive\ Request.process=10 (for RV usage)

You can modify this parameter, but it should be set to a slightly higher value than is set for the "From-DMZ" parameter set above. This parameter limits the number of outbound response jobs that can be processed.

MaxJobs.Gateway/BMH/Outbound/From-Back-Office/JMS/Response\ Receiver.process=0 (for EMS usage)

MaxJobs.Gateway/BMH/Outbound/From-Back-Office/RV/Receive\ Response.process=0 (for RV usage)

The BC EDI User's Guide has a section entitled, "Tuning for Improved Performance" which describes other parameters that can be optimized to get more performance out of the BC engine.

3). If using RV as the transport, make sure that subscribing processes are confirming messages correctly and that the ledger file size does not exceed 50 KB in operation. A continually increasing ledger file size is an indication that the messages are not being confirmed correctly by subscribers. This can lead to a large amount of RV retransmissions,which consume large amounts of memory in the BC engine.

4). If more performance is needed after resolving memory issues, consider using BC engines in a load balancing configuration. In order to do load balancing with BusinessConnect 7.X, you need to have at least two engines running (this assumes you are using EMS has the Interior Server-Private Process communication medium.   For BC 6.X, see the next paragraph).  EMS will dispatch messages to multiple engines using a randomized process.

For BusinessConnect 6.X installations, you need to have at least three engines running, as the GS only supports RV communication.  One engine functions as a scheduler for the two other engines.  You can put the scheduler engine on the same box as one of the other engines. To designate one of the engines as a scheduler, look for the following section of the deployed engine tra file.

#DMZ to Interior Queue Configuration

#tibco.clientVar.gatewayProperties/transports/Intercomponent/msh/wweight=1

#tibco.clientVar.gatewayProperties/transports/Intercomponent/msh/wtasks=10

#tibco.clientVar.gatewayProperties/transports/Intercomponent/msh/wcompletetime=0

#tibco.clientVar.gatewayProperties/transports/Intercomponent/msh/sweight=1

#tibco.clientVar.gatewayProperties/transports/Intercomponent/msh/sheartbeat=1

#tibco.clientVar.gatewayProperties/transports/Intercomponent/msh/sactivation=3.5

Uncomment the following line and set the value to a number greater than 1 .

tibco.clientVar.gatewayProperties/transports/Intercomponent/msh/sweight=10

This will force the engine to become a scheduler.