1. Determine the Kafka client library version used by your BE application. This is typically done by inspecting the $BE_HOME/lib/ext/tpcl/ folder. This folder will contain the 'kafka-clients' library, where the version will be specified as part of the filename.
- For BE 5.6.1, kafka-clients version 2.3.0 is installed
- For BE 6.2.1, kafka-clients 2.8.1 is installed
2. Download the kafka-log4j-appender JAR file from Maven Central. The version you choose should match the version determined in step 1.
3. Place kafka-log4j-appender-x.y.z.jar in your BE installation under $BE_HOME/lib/ext/tpcl/.
4. Create a custom log4j configuration file that uses the KafkaAppender.
BE 5.6.1 (log4j 1.x) example:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="console" class="org.apache.log4j.ConsoleAppender">
<param name="Target" value="System.out"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy MMM dd HH:mm:ss.SSS 'GMT'XXX} %p [%c{3}] - %m%n"/>
</layout>
</appender>
<appender name="kafkaAppender" class="org.apache.kafka.log4jappender.KafkaLog4jAppender">
<param name="brokerList" value="brokerhost:9092" />
<param name="topic" value="be_log"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n" />
</layout>
</appender>
<logger name="org.apache.kafka">
<level value="WARN"/>
</logger>
<root>
<priority value ="info" />
<appender-ref ref="console" />
<appender-ref ref="kafkaAppender" />
</root>
</log4j:configuration>
BE 6.x (log4j 2.x) example:
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%d{yyyy MMM dd HH:mm:ss.SSS 'GMT'XXX} %p [%c{3}] - %m%n" charset="UTF-8"/>
</Console>
<Kafka name="Kafka" topic="be_log">
<PatternLayout pattern="%d{yyyy MMM dd HH:mm:ss.SSS 'GMT'XXX} %p [%c{3}] - %m%n" charset="UTF-8"/>
<Property name="bootstrap.servers">brokerhost:9092</Property>
</Kafka>
</Appenders>
<Loggers>
<Logger name="com.tibco.rta.action" level="info" additivity="false">
<AppenderRef ref="STDOUT"/>
</Logger>
<Logger name="org.apache.catalina" level="info" additivity="false">
<AppenderRef ref="STDOUT"/>
</Logger>
<Logger name="org.apache.http" level="info" additivity="false">
<AppenderRef ref="STDOUT"/>
</Logger>
<Logger name="org.apache.kafka" level="WARN">
<AppenderRef ref="Kafka"/>
</Logger>
<Root level="info">
<AppenderRef ref="STDOUT"/>
<AppenderRef ref="Kafka"/>
</Root>
</Loggers>
</Configuration>
5. Configure your BE application to use this custom logging configuration, according to the Help under Configuration Guide > Collections, Agent Classes, and Processing Units > Log Configurations > Overriding the Default Logging Mode.
Important Note from Log4j KafkaAppender documentation: "Make sure to not let org.apache.kafka log to a Kafka appender on DEBUG level, since that will cause recursive logging".