When the RAM utilization hits 85% on the Node Manager, Spotfire will begin paging out data to disk according to an algorithm to optimize end-user performance. Before that, the entire cache (analyses loaded by users or by Scheduled Updates) will be maintained in RAM. This article describes that paging behavior of the Spotfire Web Player application.

When the RAM utilization hits 85% on the Node Manager, Spotfire will being paging out data to disk according to an algorithm to optimize end-user performance. Before that, the entire cache (analyses loaded by users or by Scheduled Updates) will be maintained in RAM. This article describes that paging behavior of the Spotfire Web Player application.

Page File:

When paging occurs, Spotfire will create a page file in the DataStorage Temp folder, by default located at the following.

• <Installation Directory>\tibco\tsnm\<Spotfire Version>\nm\services\WebWorker-<Service Build Version>-<Service GUID>\Temp\<Instance GUID>\DataStorage\

Or for example:

• <Installation Directory>\tibco\tsnm\7.7.0\nm\services\WebWorker-26.10.10706.2923-82bc56c3-f677-4971-b44d-4fea0626620f\Temp\6cd602dc-77c1-4fac-b67f-3efe248b3aaa\DataStorage\

The page file will have a file name comprised only of a GUID, such as 52501a1b-b32d-4729-98c0-5d05c74e6230 (no file extension).

Beginning with Spotfire 7.7, the page file is created by Spotfire in a way that allows the operating system to deallocate disk blocks and use less space for the file. So the operating system can shrink the size of the file on disk if the file system supports it. But before that, this page file will never be shrunk until the instance is restarted, even if the space is no longer needed for paging. For example, if a large analysis was closed and the data was released, the Web Player can work only in RAM again. It is important to note that unused space in the page file will be reused when needed before allocating more space for the page file, so that it will not grow more than necessary. Because of this, per the system requirements, you should ensure that there is sufficient disk space to match the amount of RAM on the system, if not more to account for all files maintained on that disk. This is critical if the system runs out of Temp disk space since all processing will stop until more space is available.

The location of this page file can be modified by editing the Spotfire.Dxp.Worker.Host.exe.config file and modifying the following node.

<setting name="TempFolder" serializeAs="String">
    <value>Temp</value>
</setting>

The default "Temp" value inside of the <value></value> tag can be updated to the desired temp folder, such as the following.

<value>E:\spotfiretemp</value>

This should always be a fast, local disk.
 

Page Management and Internal Optimization:

When the RAM utilization hits 85% on the Node Manager, the Spotfire data engine will begin paging out blocks according to a least recently used algorithm. It is optimized to maximize the performance of the system and will prioritize what is to be paged to disk. For example, it will page out data that is not recently used, not used in the analysis, or cached calculations that will be faster to page in (bring back to RAM from the page file) instead of re-calculate. Another common example is that an analysis will contain a lot of data tables with many of columns and a large portion of those columns are not needed when using the analysis. These will be paged out since this is an efficient way of minimizing RAM usage with no performance hit since they will never be paged in to RAM.

The data engine and its paging are not aware of "analyses" (i.e., which block belongs to which analysis), since an analysis is comprised of many different low level elements that are shared, managed and moved around to be as efficient as possible. Essentially, the paging is managed on a much lower level than that of the "analysis" level. Therefore, it is not possible to correlate the page file with any particular analysis and it is not possible to say which analyses are paged to disk versus maintained in RAM, since most analyses may have at least some part paged to disk.
 

Performance:

Even though there is a great deal of optimization and management going on within Spotfire to maximize performance when paging is occurring, it is always preferred to have sufficient resources to handle the total load (cached data plus interaction with active concurrent users) within RAM. If performance is a priority, then the system should be properly sized to avoid paging .

In general, the paging in of data again (from disk to RAM) is fast in most cases, but any time RAM is exhausted, the performance will not be as good as if only RAM is used. Paging in will not be able to work as well if large amounts of data are rapidly paged out and in, as is the case when there are extreme resource constraints (volume of cached, updated, and accessed reports versus available RAM). This happens if the data that is needed has to be paged out due to resource constraints. In that case, more RAM is needed on the Node Manager to be able to sufficiently host those analyses that are cached/open. 

A useful performance counter, which is captured in the PeformanceCounterlog.txt and displayed in the Monitoring & Diagnostic tab is:

• TIBCO Spotfire Data Engine.paged in bytes

This measures the accumulated bytes that were paged out to disk but then had to be pulled back in to RAM. Note that the same memory can be paged in and out many times and that will be included in the sum, meaning the values can, over time, grow much larger than the size of the page file. If this counter grows fast it means Spotfire is paging back into RAM too much and additional RAM is required on the system, based on current usage and load.

Doc: TIBCO Spotfire Server Installation and Administration Help > Manually editing the service configuration files

• https://docs.tibco.com/pub/spotfire_server/7.11.1/doc/html/TIB_sfire_server_tsas_admin_help/GUID-958A3AB5-7243-4837-9AEA-78AA274A6F7E.html