StreamBase Studio and Server use the operating system's character encoding. On Windows this may be UTF-8, or ANSI with a specific code-page such as windows-1252 (ANSI Latin 1), gb2312 (Chinese Simplified), or many others. On Linux the default character encoding is often determined by the locale, and different utilities (bash, python) may at the same time default to other encodings.

Character representation in terminal windows and log files will appear corrupted if displayed with a different encoding than was used to create them. Editing saved files in an editor with the wrong encoding will corrupt them (only characters in the ASCII range will be preserved), preventing them from being reverted to a correct representation later.

How can a consistent character encoding be configured for StreamBase?

Configuration guidance

The default character encoding for a StreamBase Studio session or new StreamBase Server instance may be overridden following the instructions here:
  TIBCO StreamBase Documentation > Authoring Guide > Miscellaneous Topics > Unicode Support

There are three configurations needed, one for Studio, one for Server String Handling, and one for Server Logging.
 

STUDIO SETTINGS

Regardless of the encoding selected, Studio saves EventFlow applications in an XML format that specifies UTF-8 encoding, which ensures that any Unicode strings you use in expressions in operators are preserved. All files created and saved by StreamBase Studio are saved as Unicode-compliant files with UTF-8 encoding. Additional settings are required for correct presentation in the Studio views.

For Studio, set the environment variable STREAMBASE_STUDIO_VMARGS in the operating system runtime environment in which Studio is launched. Example settings are:
  STREAMBASE_STUDIO_VMARGS=-Xms512m -Xmx1024m -Dstreambase.tuple-charset=UTF-8
On Microsoft Windows, set this environment variable in the System Properties, Advanced tab, Environment Variables dialog in the "User variables" list. When set, when Studio is run from a Windows menu or application icon the variable is set in Studio's runtime environment.

The memory settings "Xms" and "Xmx" must be set when setting the STREAMBASE_STUDIO_VMARGS environment variable, and for larger projects the Xmx (maximum allowed memory) setting may need to be set larger than the above example.

SERVER STRING HANDLING SETTINGS

For StreamBase 7, in the sbd.sbconf use:

<java-vm>
  <sysproperty name="streambase.tuple-charset" value="UTF-8" />
</java-vm>

For additional help, please see the documentation page referenced above.

For StreamBase 10, in the StreamBaseEngine HOCON configuration file use:

name = "myengine"
version = "1.0.0"
type = "com.tibco.ep.streambase.configuration.sbengine"
configuration = {
  StreamBaseEngine = {
    systemProperties = {
      "streambase.tuple-charset" = "UTF-8" 
    }
  }
}

Place the HOCON configuration file in the src/main/configurations folder.

These settings only affect how string values are interpreted within the server and do not affect log output or console representation.

SERVER LOGGING SETTINGS

The logging character encoding is controlled by the Logback configuration. The default logging configuration which writes to the console (STDOUT and STDERR) uses the operating system's default character encoding. To override this, provide your own logback.xml configuration.

This example logback.xml configuration writes correct UTF-8 characters to the console and a named log file in the node logs-folder location:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="RootConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <charset>UTF-8</charset>
      <pattern>%d{yyyy-MM-dd HH:mm:ss.}%usecs%d{Z} [%process:%thread] %-5level %logger: %msg%n</pattern>
    </encoder>
  </appender>

  <appender name="utf8logfile" class="ch.qos.logback.core.FileAppender">
    <file>${com.tibco.ep.dtm.logging.logDirectory}/${com.tibco.ep.dtm.logging.logFileNamePrefix}_utf8.log</file>
    <encoder>  
      <charset>UTF-8</charset>
      <pattern>%d{yyyy-MM-dd HH:mm:ss.}%usecs%d{Z} [%process:%thread] %-5level %logger: %msg%n</pattern>
    </encoder>
  </appender>
  
  <root>
    <level value="info"/>
    <appender-ref ref="RootConsoleAppender" />
    <appender-ref ref="utf8logfile" />
  </root>
</configuration>

For StreamBase 7, place the logback.xml file in the project top-level folder.

For StreamBase 10, place the logback.xml file in the src/main/resources folder.