Skip to content

What is Burrow

Jump to bottom
toddpalino edited this page Nov 3, 2015 · 8 revisions

#Overview Burrow is a monitoring tool for keeping track of consumer lag in Apache Kafka. It is designed to monitor every consumer group that is committing offsets to either Kafka or Zookeeper, and to monitor every topic and partition consumed by those groups. This provides a comprehensive view of consumer status.

Burrow also provides several HTTP request endpoints for getting information about the Kafka cluster and consumers, separate from the lag status. This can be very useful for creating applications that assist with managing your Kafka clusters when it is not convenient (or possible) to run a Java Kafka client.

#Why Not MaxLag? The standard Kafka consumer does have a built-in metric to track MaxLag. While this can be convenient, it has several flaws:

  • MaxLag must be monitored on every consumer The MaxLag metric must be collected from every consumer. These metrics have to be collated and interpreted separately.
  • MaxLag is only valid when the consumer is live The metric is reported by the consumer itself. If ithe consumer is not running, no metric is available.
  • MaxLag is not objective Because the consumer itself reports the metric, MaxLag cannot be an objective measure of consumer lag. The consumer measures it after fetching messages, so if there is any problem with consuming, an incorrect value can be reported.
  • MaxLag is only provided by the Java client The only official Kafka client is the Java client, and this is the only client that has the metric available. It can certainly be worked into other clients, but then you have to worry about subtle differences in measurement and collection of the metric.

#How Does It Work? Burrow runs a Kafka client that consumes the __consumer_offsets topic, which is a special internal topic where all consumer offset commits are stored. This provides a stream of every consumer's status for consumers that commit offsets to Kafka. It also periodically scans the consumers tree in Zookeeper and gathers offsets from consumers that commit offsets there. Additionally, Burrow periodically queries all the brokers in the cluster for the current HEAD offset (the most recent offset). Upon demand, via the HTTP endpoint, or on a set interval determined by the , the status of each consumer group is calculated to determine if it is OK or having problems.

For more information about the specific rules on evaluation, review the Consumer Lag Evaluation Rules page.

Clone this wiki locally