When there is a situation where all Web Players need to be restarted, after the restarts the load on the instances of the Web Players may became far from evenly spread. Even after restarting Instances and Web Players the system may not redistribute the load resulting in long queues, or strained/exhausted instances. While some Web Players will have a large load, and other Web Players remain idle which may result in long loading and failing scheduled dashboards.

Cause: If there are many active scheduled updates rules that are routed to a set of workers in a resource pool and these workers are restarted, there is a risk that all of the Web Players will not be available at the time when the set of rules is evaluated again by the Spotfire Server(s). If that happens, the router will choose only between those Web Players that are already available. When the routing is done, it will not change even if there is a larger number of unused available workers at a later time.

All

Below are three suggestions to help avoid this undesired situation: One or more of them can be used:

Option 1: Enable failover-delay
Instructions to enable the failover-delay in the Spotfire Server configuration. See Manually editing the server configuration in an XML or text editor for more details on this process:
    1. Export the currently used configuration from the Spotfire database. 
    2. To export, either login to the uiconfig.bat file (uiconfig.sh on Linux) which is located in the <Installation dir>\tomcat\spotfire-bin directory  OR  execute CLI command export-config  
    3. Open exported server configuration from the given location in the text or XML editor and search for deployment
    4. Change the value here in the below section to true:
        <!-- If enabled, introduces a delay to before scheduled updates are loaded when a service starts or stops.
        If disabled, scheduled updates are reevaluated immediately. -->
        <failover-delay>
        <enabled>true</enabled>
        <!-- delay expressed in seconds -->
        <delay>300</delay>
        </failover-delay>
    5. Save the changes and import the configuration back to the database, either using the uiconfig tool or use the import-config command
    6. Restart the Spotfire Server


Option 2: Temporarily disabling rules
If all the Web Players have to be restarted, before doing that, use the following steps:

1.     Disable all scheduled updates rules
2.     Restart the Web Players
3.     When all workers are up, enable the rules again

Option 3: Stagger the Scheduled Update rules
For those rules where it is possible: add a gap in the schedule when the analyses will be unloaded and then loaded again

This article describes the cause and the possible suggestions which will help to avoid imbalanced load distribution between the Web Player instances after the restart of all Web Players.

Doc: export-config 

•  https://docs.tibco.com/pub/spotfire_server/12.0.4/doc/html/TIB_sfire_server_tsas_admin_help/server/topics/export-config.html

Doc: import-config

• https://docs.tibco.com/pub/spotfire_server/12.0.4/doc/html/TIB_sfire_server_tsas_admin_help/server/topics/import-config.html

Doc: Manually editing the server configuration in an XML or text editor  

• https://docs.tibco.com/pub/spotfire_server/12.0.4/doc/html/TIB_sfire_server_tsas_admin_help/server/topics/manually_editing_the_server_configuration_in_an_xml_or_text_editor.html