Resolution:
In our tests and in production environments, there are some configuration changes that can improve performance on large grids (those > 1,000 Engines). This will improve Manager performance in situations, for example, when the Broker Monitor response time may take several seconds to update.

The following changes can improve performance on large Grids:


    * On the Brokers, the default value for Engine "Max Millis Per Heartbeat" is 2 minutes. Try to keep the aggregate mean heartbeats per second (num engines * heartbeat frequency) to 40 or less. To meet this goal you may need to increase the value of "Max Millis Per Heartbeat" to greater than 2 minutes. To change this setting, click Manager, then click Manager Configuration, then click Comminication; the setting is under the Engine Heartbeat heading.
    * Decrease the SSL "Token Timeout," which is actually in effect regardless of SSL, for both the "Broker Resources" and "Director Resources" to be 5 minutes. The settings are on the Manager Configuration page, in the Security section, under the Resource Deployment heading.
    * Increase the Assignment Timeout, on the Manager Configuration page, in the Services section, to 60000 ms. Increasing this allows more time for an Engine to connect and pickup an assigned task when the Broker is under heavy load. This value should be increased if you see 'Task assignment expired:'… messages often.
    * Separate the administrative port (web browser port) from the HTTP messaging port, to all administrative access even under heavy messaging load. This is explained in KB article #235.
    * On the Manager Configuration page, in the Communication section, change Messaging Retry Wait to at least 5000 ms or even higher; change Maximum Resource Download Connections to 50; change Driver/Engine/Daemon Socket Timeout to 120 seconds; make sure All Sender Socket Timeout is set to 5000; change Driver Heartbeat Timeout Factor to 15; change Engine Heartbeat Timeout Factor to 4. In the services section, Task Reservation Timeout to 120, and Assignment Timeout(ms) to 60000.
    * Increase the heap size. See KB article #8 - "The Manager runs out of memory" for details.
    * Add or change timeouts for Broker and Secondary Director connections. Sender Socket Timeout properties are exposed on the Manager Configuration page, in the communications section, under the Broker Connections and Secondary Director Connections. The default value is 5000 milliseconds.