It's generally doable to mix Windows and Linux Operating Systems in the Spotfire Server cluster, but this can add complexity as some tasks need to be done twice. It's usually recommended to standardize on one OS and then handle exceptions accordingly. For example, if Linux is chosen as the default OS, then special cases like old analysis files that rely on customer extensions only available on Windows can be addressed. Vice versa, if Windows is the default, selecting Linux may bring advantages like Docker support for data function services.
For Spotfire Server Service it is possible to mix Operating Systems but will not work completely out of the box as the import and export folder for the library cannot be shared between Windows and Linux as they use different path styles. So you would need to specify different import/export folders for the different OS/types but not recommend mixing Spotfire Server OS types. However, you can just use overrides for the "other" OS (when using set-config-prop) to the path.
The best approach for Web Player and Automation Services depends on what the customer is trying to achieve. While it works, there are differences in how Linux/Netcore works compared to the Windows/Net framework. Additionally, all customer C# extensions need to be rebuilt, and other drivers need to be installed. However, this can be managed with resource pools and routing rules. For instance, if a Spotfire dashboard uses something that is not available on the Linux Web Player, it can be routed to the Windows Web Player.
Know the limitations of using Linux for Web Player and Automation Services on the Linux O.S.
https://docs.tibco.com/pub/spotfire_server/latest/doc/html/TIB_sfire_server_tsas_admin_help/server/topics/running_web_player_or_automation_services_on_linux.html
Note: There are some differences when using Web Player or Automation Services on Linux, compared to Windows.
Best Approach?
Typically the one where the admins are comfortable configuring, monitoring, and doing admin tasks on the O.S.
But in general, enough resources and only a single service per machine.