Managing Kafka Failover in TIBCO Streaming

Managing Kafka Failover in TIBCO Streaming

book

Article ID: KB0072556

calendar_today

Updated On:

Products

TIBCO Streaming

Description

I have added my main Kafka broker and two backup brokers to the Kafka Consumer "Brokers" list. When the first broker becomes unavailable, the adapter does not connect to any of the other brokers. What is the correct configuration for failover?

Issue/Introduction

A Kafka cluster requires replicated topics to enable failover when using the TIBCO Streaming Kafka adapters.

Resolution

In this case, the reason the Kafka Consumer adapter did not connect to another broker is because there were too few brokers remaining in the cluster to support the subscribed topic's replication count. The Kafka cluster could no longer serve that topic.

The broker list is not designed to support fail-over functionality, but only discovery. Failover is dependent only on how a topic's replication count is configured and the number of available Broker processes. The Kafka adapter (client) "Brokers" property identifies at least one live broker which is used to identify the Zookeeper. The zookeeper discovers all the brokers and which topics are available. It then reports that back to the client. In this way, the Broker list is effectively a way for the adapter to find the zookeeper. When subscribing to a topic the zookeeper will then inform the client which broker to connect to to receive updates from that topic. Once the client has contacted the zookeeper, then if an assigned broker for a subscription becomes unavailable, the zookeeper informs the client the next broker to connect to (if any) for that subscription.

If replication is set to '1' for a topic, that topic is stored in the logs of a single broker. If that broker becomes unavailable, the topic is inaccessible. There is no failover setting to correct this through the Kafka API. Kafka instead supports replicated topics across multiple brokers. Setting the replication count to '2' will allow fail-over to happen if one broker becomes unavailable. The zookeeper will inform the client the next broker to use (even if the next broker does not appear on the adapter's "Brokers" list). In the set of brokers known to the zookeeper there need to be enough live brokers to support a topic's configured replication count. For example, you cannot create a topic with replication set to '2' if there's only one broker running. Similarly, clients do not get to say to which broker they will use to access a topic.

The correct failover configuration for the Kafka adapters is to:
  1. Subscribe to a topic which has a replication count greater than '1', and
  2. Specify one or more brokers so that the adapter can find the zookeeper when connecting or subscribing.
Using this configuration, when fail-over needs to happen, as long as any broker which is known to the Zookeeper has a replica of the topic then the new broker and topic will be reported to the adapter by the zookeeper.

The Kafka Consumer and Producer adapters provide an optional Command input port that allows the application to send in a new list of Brokers using the "updateBrokers" command. This allows the application to, at any time, override the statically defined adapter "Brokers" property value. The Kafka Admin adapter may be used to query the available brokers using its Command input port and Brokers output port to provide a new list of brokers to the other Kafka adapters.