Resolution:
GridServer 4.2 has the best scalability of any GridServer release to date. In our tests and in production environments, there are some configuration changes that can improve performance on large grids (> 1,000 Engines). This will improve Manager performance in situations, for example, when the Broker Monitor response time may take several seconds to update.

The following changes can improve performance on large Grids:

    * We strongly suggest that you update to GridServer 4.2 update 15, which includes the upgrade of Resin 3.0.15 detailed below.
    * Upgrade to Resin 3.0.15. GridServer currently uses Resin 3.0.14. An upgrade package can be obtained from the DataSynapse customer support team.
    * On the Brokers, increase the Engine "Max Millis Per Heartbeat" value to be at least 2 minutes, or even higher. The default is 30 seconds. Try to keep the aggregate mean heartbeats per second (num engines * heartbeat frequency) to 40 or less.To change this setting, click Manager, then click Manager Configuration, then click Comminication; the setting is under the Engine Heartbeat heading.
    * Decrease the SSL "Token Timeout," which is actually in effect regardless of SSL, for both the "Broker Resources" and "Director Resources" to be 5 minutes. The settings are on the Manager Configuration page, in the Security section, under the Resource Deployment heading.
    * Increase the Assignment Timeout, on the Manager Configuration page, in the Services section, to 60000 ms. Increasing this allows more time for an Engine to connect and pickup an assigned task when the Broker is under heavy load. This value should be increased if you see 'Task assignment expired:'… messages often.
    * Separate the administrative port (web browser port) from the HTTP messaging port, to all administrative access even under heavy messaging load. This is explained in KB article #235.
    * On the Manager Configuration page, in the communication section, change Messaging Retry Wait to 10000 ms; change Maximum Resource Download Connections to 50; change Driver/Engine/Daemon Socket Timeout to 120 seconds; make sure All Sender Socket Timeout is set to 5000 (available only in GridServer 4.2 update 7 or later); change Driver Heartbeat Timeout Factor to 15; change Engine Heartbeat Timeout Factor to 4. In the services section, Task Reservation Timeout to 120, and Assignment Timeout(ms) to 60000.
    * Increase the heap size. See KB article #8 - "The Manager runs out of memory" for details.
    * Add or change timeouts for Broker and Secondary Director connections. In GridServer 4.2 Update 7 and later, Sender Socket Timeout properties are exposed on the Manager Configuration page, in the communications section, under the Broker Connections and Secondary Director Connections. The default value is 5000 milliseconds.

Prior to Update 7, you will need to change every Directors' director.xml files. Locate the ServerSocketConnectionManager block for Broker Connections and Secondary Director Connections, and add a timeout to a non-zero value. For example:

&ltconnectionmanager class="ServerSocketConnectionManager" name="Broker Connections">
    &ltproperty name="address" value="$DSAddress$"/>
    &ltproperty name="SSL" value="False"/>
    &ltproperty name="permanentConnection" value="true"/>
    &ltproperty name="timeout" value="5000"/>
</connectionmanager>
&ltconnectionmanager class="ServerSocketConnectionManager" name="Secondary Director Connections">
    &ltproperty name="address" value="$DSAddress$"/>
    &ltproperty name="permanentConnection" value="true"/>
    &ltproperty name="SSL" value="False"/>
    &ltproperty name="timeout" value="5000"/>
</connectionmanager>