How to configure a KafkaAppender to send log messages from your BuinsessEvents application to a Kafka topic

Products	Versions
TIBCO BusinessEvents Enterprise Edition	5.6.1, 6.x

Description

You may use the log4j KafkaAppender to send messages from your BusinessEvents application console to a Kafka topic.

Issue/Introduction

Outlines the basic steps needed to configure a log4j KafkaAppender to send BusinessEvents log messages to a Kafka topic.

Environment

All Supported Platforms

Resolution

1. Determine the Kafka client library version used by your BE application. This is typically done by inspecting the $BE_HOME/lib/ext/tpcl/ folder. This folder will contain the 'kafka-clients' library, where the version will be specified as part of the filename.

For BE 5.6.1, kafka-clients version 2.3.0 is installed
For BE 6.2.1, kafka-clients 2.8.1 is installed

2. Download the kafka-log4j-appender JAR file from Maven Central. The version you choose should match the version determined in step 1.

3. Place kafka-log4j-appender-x.y.z.jar in your BE installation under $BE_HOME/lib/ext/tpcl/.

4. Create a custom log4j configuration file that uses the KafkaAppender.

BE 5.6.1 (log4j 1.x) example:

<?xml version="1.0" encoding="UTF-8"?> 
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
  <appender name="console" class="org.apache.log4j.ConsoleAppender"> 
    <param name="Target" value="System.out"/> 
    <layout class="org.apache.log4j.PatternLayout"> 
      <param name="ConversionPattern" value="%d{yyyy MMM dd HH:mm:ss.SSS 'GMT'XXX} %p [%c{3}] - %m%n"/> 
    </layout> 
  </appender> 
  <appender name="kafkaAppender" class="org.apache.kafka.log4jappender.KafkaLog4jAppender">      
      <param name="brokerList" value="brokerhost:9092" />
      <param name="topic" value="be_log"/>
      <layout class="org.apache.log4j.PatternLayout">
         <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n" />
      </layout>
   </appender>
 <logger name="org.apache.kafka">
    <level value="WARN"/>
</logger>  
  <root> 
    <priority value ="info" /> 
    <appender-ref ref="console" />
    <appender-ref ref="kafkaAppender" />    
  </root>  
</log4j:configuration>

BE 6.x (log4j 2.x) example:

<Configuration>
  <Appenders>
    <Console name="STDOUT" target="SYSTEM_OUT">
      <PatternLayout pattern="%d{yyyy MMM dd HH:mm:ss.SSS 'GMT'XXX} %p [%c{3}] - %m%n" charset="UTF-8"/>
    </Console>
    <Kafka name="Kafka" topic="be_log">
      <PatternLayout pattern="%d{yyyy MMM dd HH:mm:ss.SSS 'GMT'XXX} %p [%c{3}] - %m%n" charset="UTF-8"/>
        <Property name="bootstrap.servers">brokerhost:9092</Property>
    </Kafka>
  </Appenders> 
  <Loggers>
  <Logger name="com.tibco.rta.action" level="info" additivity="false">
            <AppenderRef ref="STDOUT"/>
        </Logger>
  
  <Logger name="org.apache.catalina" level="info" additivity="false">
            <AppenderRef ref="STDOUT"/>
        </Logger>
    <Logger name="org.apache.http" level="info" additivity="false">
            <AppenderRef ref="STDOUT"/>
        </Logger>
    <Logger name="org.apache.kafka" level="WARN">
            <AppenderRef ref="Kafka"/>
        </Logger>
    <Root level="info">
      <AppenderRef ref="STDOUT"/>
      <AppenderRef ref="Kafka"/>
    </Root>
  </Loggers>
</Configuration>

5. Configure your BE application to use this custom logging configuration, according to the Help under Configuration Guide > Collections, Agent Classes, and Processing Units > Log Configurations > Overriding the Default Logging Mode.

Important Note from Log4j KafkaAppender documentation: "Make sure to not let org.apache.kafka log to a Kafka appender on DEBUG level, since that will cause recursive logging".

Welcome to "KB Articles"