How to fix special character conversion errors when importing data into Spotfire Statistica from Databricks.
Article ID: KB0138570
Updated On:
| Products | Versions |
|---|---|
| Spotfire Statistica | 14.1 |
Description
Databricks natively processes and exports data using UTF-8 encoding. If the local Windows environment or client application attempting to ingest the data is set to a legacy local code page (like ANSI or Latin-1), it will fail to map the incoming UTF-8 special characters correctly, resulting in a conversion failure.
Screenshot of the Databricks data that we want to import into Statistica.
Screenshot of the data that was imported from Databricks to Statistica before the fix.
Environment
Windows(OS)
Resolution
Steps to Resolve:
1. Open the Windows Control Panel on the machine performing the import.
2. Click on "Clock and Region", then select "Region".
3. In the Region dialog box, navigate to the "Administrative" tab.
4. Under the 'Language for non-Unicode programs' section, click the "Change system locale..." button.
5. In the settings window that appears, locate and check the box next to:
"Beta: Use Unicode UTF-8 for worldwide language support".
6. Click "OK" to apply the changes.
7. Restart your computer for the system-wide encoding changes to take effect.
8. Re-run the Databricks data import process. The special characters will now convert correctly.
Issue/Introduction
When importing data from Databricks into a Windows-based system or application, special characters (such as special characters like < > or other symbols) may fail to convert properly, causing data corruption or import errors.
