Consumer Kafka Zookeeper

The "kafka_zk" consumer module connects to the Zookeeper ensemble that the Kafka cluster uses to store metadata and monitors the offsets that are committed to the /consumers tree. It does this by setting a watch on the list of consumers, the list of topics for each consumer, the list of partitions for each topic, and the offset for each partition. When any information changes, the module transmits the updates to the Storage subsystem for use in group status evaluations. This handles all older consumers that are committing offsets to Zookeeper.

Configuration

The subheading must be a unique name, such as [consumer.myconsumers]. The "myconsumers" part is the name that will be used when logging.

[consumer.myconsumers]
class-name="kafka_zk"
cluster="myclustername"
servers=[ "zk01.example.com:2181", "zk02.example.com:2181", "zk03.example.com:2181" ]
zookeeper-timeout=30
zookeeper-path="/kafka-cluster"

Key	Value Type	Required	Default Value	Description
class-name	string	yes	(empty)	This is the name of the consumer module type. For this module, it must be "kafka_zk"
cluster	string	yes	(empty)	This is the name of the cluster to which the consumers in this module belong.
servers	list of hostname:port	yes	(none)	A list of strings in the form "hostname:port" that point to the servers in the Zookeeper cluster. At least one is required.
zookeeper-timeout	integer	no	30	The session timeout to use when connecting to Zookeeper.
zookeeper-path	string	no	(empty)	If specified, this is the root of the Kafka cluster metadata in the Zookeeper ensemble. If not specified, the root path is used.
group-allowlist	string (regex)	no	(empty)	If specified, only send offsets for groups that match this regular expression.
group-denylist	string (regex)	no	(empty)	If specified, only send offsets for groups that DO NOT match this regular expression. This is processed after the allowlist (if specified).