How can I implement a custom logging configuration for my Streaming application? The default configuration rolls the log files too quickly, so I am not able to see the complete log output.
 

Provides a sample logback.xml configuration file and describes the steps needed to use this configuration in a Streaming application.

1. Stop your Streaming application if it is currently running.

2. Create a logback.xml file with a RootFileAppender and RootConsoleAppender. For example..
 

<?xml version="1.0" encoding="UTF-8"?>

<configuration scan="true">
  <appender name="RootFileAppender" 
            class="ch.qos.logback.core.rolling.RollingFileAppender">
    <append>true</append>
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
      <level>info</level>
    </filter>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>sample-logging-logback-%d{yyyy-MM-dd}.log.gz</fileNamePattern>
    </rollingPolicy>
    <encoder>
      <pattern>%date{yyyy-MM-dd HH:mm:ss} [LogBack %level] %logger{0} - %msg%n</pattern>
    </encoder>
  </appender>
  <appender name="RootConsoleAppender" 
            class="ch.qos.logback.core.ConsoleAppender">
      <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
      <level>info</level>
    </filter>
    <encoder>
      <pattern>[LogBack %level] %logger{0} - %msg%n</pattern>
    </encoder>
  </appender>
  <root>
    <level value="info"/>
    <appender-ref ref="RootFileAppender"/>
    <appender-ref ref="RootConsoleAppender"/>
  </root>
</configuration>

3. Add this logback.xml to your Streaming project under the sub-folder: src/main/resources/. You can also place it in the src/test/resources folder if you need to run Maven tests.

4. Restart your Streaming application.

NOTE 1: The above example uses a TimeBasedRollingPolicy without any custom options set. That means the log files will grow unbounded. This configuration prevents the logs from getting rolled too quickly (so you don't lose any information), but it can cause disk space issues if the logs are not manually deleted after some time. For customizing the options for a TimeBasedRollingPolicy (like maxHistory and totalSizeCap), refer to the Logback documentation (under the sub-heading 'TimeBasedRollingPolicy').

NOTE 2: In general, it is not recommended to log details about data (which your Streaming application is processing) to the Streaming engine log. Not only is some of this information potentially private/sensitive, it is also a common cause of logs rolling too quickly. If this data needs to be preserved (for later analysis), it is recommended to direct those log messages to a separate file. This can be done by adding an additional logger/appender to your custom Logback configuration, or by using the File Writer Output adapter.

Home > StreamBase Admin Guide > StreamBase Logging > Using StreamBase Logging