Resolution:
Internally the file adapter uses UTF-16 encoding to perform processing on the data.

No matter what the file encoding  (LATIN_1, SHIFT_JIS, UTF_8) the user chooses, the file adapter reader will convert the contents of the file it reads to UTF-16 and then perform the appropriate processing.

During publication the adapter will convert the UTF-16 data it has internally into UTF-8 or ISO8859-1 and published using TIBCO Rendezvous.

Similarly, the subscriber first converts, UTF-8 (or ISO8859-1) to UTF-16 and then depending on the file encoding it again coverts UTF-16 in to the appropriate encoding and writes it to the output file.

In order to do the conversion from one encoding to the other (SHIFT_JIS to UTF-16 or UTF-16 to UTF-8) the file adapter uses the environment variable TIB_ICU_DATA, which points to the directory where a file called "tibicudata.dat" is kept, this file is essentially a group of tables which provides mapping from one encoding to the other.

Note: UTF8 or ISO8859-1 is selected based on the repository setting, which the user selects using the repository finder.
Note: ISO8859-1 should be selected only if file encoding is LATIN_1. UTF-8 can be selected for any encoding (LATIN_1, SHIFT_JIS... etc).