Why does the cache process get stuck or throw an error when caching Japanese characters (or multi-byte characters) in MS SQL Server?

Why does the cache process get stuck or throw an error when caching Japanese characters (or multi-byte characters) in MS SQL Server?

book

Article ID: KB0137552

calendar_today

Updated On:

Products Versions
TIBCO Data Virtualization 8.8.0

Description

Microsoft SQL Server supports Japanese and other Unicode characters only with the NVARCHAR data type. Using VARCHAR can result in character loss or corruption when handling non-Latin scripts.

Reference: StackOverflow – Inserting Japanese characters into SQL Server

Environment

All supported environmnet

Resolution

1. Instead of letting TDV create the cache target table automatically, manually create the table in SQL Server with all character columns defined as NVARCHAR.

TDV user can do this by:

  • Generating the DDL for the cache target table in TDV Studio.
  • Modifying all relevant columns from VARCHAR to NVARCHAR.
  • Executing the modified DDL directly on the SQL Server.

2. Once the table is created:

  • Introspect it into TDV Studio.
  • Use this introspected table as the Target Cache Table.

Issue/Introduction

When caching data from TIBCO Data Virtualization to Microsoft SQL Server, users may encounter issues with Japanese or other multi-byte characters not displaying correctly. This is typically due to the use of the VARCHAR data type, which does not support Unicode characters such as Japanese.

This article outlines the recommended approach to ensure proper handling of Japanese characters using the NVARCHAR data type in SQL Server.