What are the data size limitations for various input files, such as CSV and XLSX files?

1. Will it return an error if we try to load a file with more rows than the upper limit into Spotfire Analyst?
2. If there is a guideline for the number of rows or amount of data and the corresponding RAM performance (for example, 100,000 rows (or 500MB) require XX performance).

• Spotfire has a limit on the number of rows it can be up to 2GB i.e. 2147483647 (two billion one hundred forty-seven million four hundred eighty-three thousand six hundred forty-seventh).
• The reason for the 2 GB limit is because row numbers are Integers (32-bit signed).
• Note: This is per the Data Table.
• Note: When it comes to Information Links, there is no specific maximum limit. This is because the information service fetches data from a database in smaller chunks and continues to retrieve the data as needed.
• Specific file formats could have limitations.

For example,

• An Excel (.xlsx) file has a limit on the Total number of rows and columns on a worksheet i.e. 1,048,576 rows by 16,384 columns. (reference: https://support.microsoft.com/en-gb/office/excel-specifications-and-limits-1672b34d-7043-467e-8e27-269d656771c3)
• A CSV file has no limit on the number of data records that can be contained in a single file. However, there are limitations with the software that you use to open and edit your CSV file. If you are opening CSV in Spotfire then the limit would be 2147483647 rows. (reference: https://csvloader.com/csv-guide/understanding-csv-and-row-limits#:~:text=In%20the%20case%20of%20CSV,that%20is%20separated%20by%20delimiters.&text=Here%20you%20will%20encounter%20a,not%20seeing%20all%20the%20data.&text=Similar%20to%20Excel%2C%20in%20Numbers,your%20file%20exceeds%201%2C000%2C000%20lines.)

Notes:

• Spotfire Analyst Tool: Should have better RAM to handle/load the data into the Analyst tool, it will consume your local system(laptop/computer) memory, otherwise, you may face difficulties when trying to open large files. The memory used also depends on the datatype e.g. string will use more RAM than numeric.
• Node Manager: There is no installation and handling limit on the web player too other than specified limits, again it is your node manager machine that should have enough RAM size/resources to process the data.
   

Q: Will it return an error if we try to load a file with more rows than the upper limit into Spotfire Analyst?
A: Opening a file with slightly more than 2147483647 rows, will not throw any error or warning but Spotfire will cap it to 2147483647 rows.

Q: If there is a guideline for the number of rows or amount of data and the corresponding RAM performance (for example, 100,000 rows (or 500MB) require XX performance)?
A: There is no simple guideline because it depends on how you visualize and work with the data. Also, Spotfire will by design try to use much of the available RAM (for caching calculations), most of which can be dropped when the amount of free memory gets low. The memory used also depends on the datatype e.g. string will use more RAM than numeric.

Microsoft Doc:

• https://support.microsoft.com/en-gb/office/excel-specifications-and-limits-1672b34d-7043-467e-8e27-269d656771c3

Spotfire Doc:

• https://docs.tibco.com/pub/sfire-analyst/latest/doc/html/en-US/TIB_sfire-analyst_UsersGuide/ncfe/ncfe_data_types.htm?rhhlterm=real&rhsyns=%20

CSV File:

• https://csvloader.com/csv-guide/understanding-csv-and-row-limits#:~:text=In%20the%20case%20of%20CSV,that%20is%20separated%20by%20delimiters.&text=Here%20you%20will%20encounter%20a,not%20seeing%20all%20the%20data.&text=Similar%20to%20Excel%2C%20in%20Numbers,your%20file%20exceeds%201%2C000%2C000%20lines.