Connection to a Kafka topic with multiple partitions

Connection to a Kafka topic with multiple partitions

book

Article ID: KB0073732

calendar_today

Updated On:

Products Versions
TIBCO Streaming 7.7.3+

Description

The Kafka Consumer adapter now (as of StreamBase 7.7.3) offers the user options to use Consumer Groups for load balancing, as described in the Kafka documentation (https://kafka.apache.org/documentation/).

Issue/Introduction

Using Kafka Consumer groups

Resolution

To use consumer group balancing, you need to make sure to subscribe with the integer 'partition' field set to null.  For example..

Command: subscribe
topic: topicname
partition: null ( If this value is null, a consumer group is created and this subscription is assigned one or more partitions by the server.)

Add to the adapter configuration, Advanced Options tab, Advanced Config the setting:
  group.id
with a string value identifying the group that this adapter belongs to. When two or more Kafka adapters connect and belong to the same group, Kafka automatically assigns each adapter a subset of the available partitions, resulting in load-balanced distribution.

Note: you cannot subscribe multiple times to the same topic with the same adapter instance.  There is code in the adapter to prevent this from happening, as it can cause issues internally with respect to tracking.  If you use a second adapter instance, it will work, however you will need to update settings for each individual adapter instance when you need to make application changes.  To avoid this, you may use a single adapter instance that other *.sbapp modules connect to.