Connection to a Kafka topic with multiple partitions
book
Article ID: KB0073732
calendar_today
Updated On:
Description
The Kafka Consumer adapter now (as of StreamBase 7.7.3) offers the user options to use Consumer Groups for load balancing, as described in the Kafka documentation (https://kafka.apache.org/documentation/).
Issue/Introduction
Using Kafka Consumer groups
Resolution
To use consumer group balancing, you need to make sure to subscribe with the integer 'partition' field set to null. For example..
Command: subscribe
topic: topicname
partition: null ( If this value is null, a consumer group is created and this subscription is assigned one or more partitions by the server.)
Add to the adapter configuration, Advanced Options tab, Advanced Config the setting:
group.id
with a string value identifying the group that this adapter belongs to. When two or more Kafka adapters connect and belong to the same group, Kafka automatically assigns each adapter a subset of the available partitions, resulting in load-balanced distribution.
Note: you cannot subscribe multiple times to the same topic with the same adapter instance. There is code in the adapter to prevent this from happening, as it can cause issues internally with respect to tracking. If you use a second adapter instance, it will work, however you will need to update settings for each individual adapter instance when you need to make application changes. To avoid this, you may use a single adapter instance that other *.sbapp modules connect to.
Feedback
thumb_up
Yes
thumb_down
No