Kafka Scala API - Advanced Nagios Plugin / CLI Tool with Kerberos support

git.io/nagios-plugin-kafka

Kafka 0.9+ API CLI Tester & Advanced Nagios Plugin with Kerberos support, written in Scala.

Tested on Hortonworks HDP 2.4.0 with Kerberos + Ranger ACLs and Apache Kafka 0.8.x / 0.9.0.1 docker images with regular ACLs.

You may need to change the Kafka library version in pom.xml / build.sbt / build.gradle before building to match your deployed Kafka server / cluster otherwise it may hang when run due to version / protocol mismatch.

See The Advanced Nagios Plugins Collection for many more related enterprise monitoring programs.

Hari Sekhon

Cloud & Big Data Contractor, United Kingdom

(ex-Cloudera, former Hortonworks Consultant)

(you're welcome to connect with me on LinkedIn)

Intro

This project builds a single self-contained Java jar file with all dependencies included and can simply be run on the command line with full switch option support:

java -jar check_kafka.jar --help

and there is an optional convenience shell wrapper script at the top level to make commands shorter:

./check_kafka --help

Run against one or more Kafka brokers, comma separated:

$ ./check_kafka --brokers localhost:9092 --topic test
OK: Kafka broker successfully returned unique message via topic 'test' partition '0', write time = 0.185s, read time = 0.045s, total time = 1.729s | write_time=0.185s read_time=0.045s total_time=1.729s

Use the --verbose switch to also show the brokers list that were tested. If you have specified one of the kerberos switches (or edited the consumer/producer properties files to do so) then the output will additionally contain the marker with sasl authentication to let you know that it was a secure configuration that was tested (originally I called this with kerberos but technically it may not be in future).

OK: Kafka broker '<hortonworks_host>:6667' successfully returned unique message via topic 'topic3' partition '0' with sasl authentication, write time = 0.148s, read time = 0.043s, total time = 0.691s | write_time=0.148s read_time=0.043s total_time=0.691s

Kafka 0.9+ API Caveats

This program only supports Kafka 0.9+ as the API changed (again) and Kerberos security was only added in the 0.9 API. For Kafka versions before 0.9 you can find Python and Perl versions of this program in the Advanced Nagios Plugins Collection that support 0.8 onwards (they dosn't support Kafka <= 0.7 as the API changed in 0.8 too and the underlying libraries in those languages don't support Kafka <= 0.7).

It appears that several errors are caught too early in the new Kafka Java API and result in embedded looping retry behaviour on encountering errors (visible in debug level logging of the base library).

I haven't found a great way of handle that behaviour as it's not exposed to the client code so it ends up being handled via my generic default self timeout mechanism that I apply to all my tools. Hence if you specify an incorrect --brokers <host>:<port> or the Kafka brokers are down or you fail to negotiate the protocol due to security settings you will only receive a generic UNKNOWN: self timed out after 10 secs message as the code self terminates.

Otherwise the Kafka API would just hang there indefintely as it keeps retrying deeper in the library. I've tried various settings to get it to time out but nothing worked and I even posted to the Kafka users mailing list without an answer. If you know of a setting that will make the Kafka Client library time out and return the more specific error then please let me know and I'll update this code accordingly.

Kerberos Support

See the conf/ directory for JAAS kerberos configurations.

If you're running the code on a Hortonworks Kafka broker it'll auto-detect the HDP configuration and use that.

Build

Quick Start - Docker

A Dockerized pre-built version is available on DockerHub.

If you have docker installed this one command will download and run it:

docker run harisekhon/nagios-plugin-kafka check_kafka --help

Automated Build from Source

curl -L https://git.io/nagios-plugin-kafka-bootstrap | sh

OR

Maven, Gradle and SBT automated builds are all provided.

A self-contained jar file with all dependencies will be created and symlinked to check_kafka.jar at the top level.

The Maven and Gradle builds are best as they will auto bootstap and run with no prior installed dependencies other than Java and make to kick it off.

The default make build will trigger a Gradle bootstrap from scratch with an embedded checksum for security:

make

You can call any one of the 3 major build systems explicitly instead, which will recurse to build the library submodule using the same mechanism:

Maven:

make mvn

Gradle:

make gradle

SBT:

make sbt

Custom TLDs

If using bespoke internal domains such as .local, .intranet, .vm, .cloud etc. that aren't part of the official IANA TLD list then this is additionally supported via a custom configuration file lib/resources/custom_tlds.txt containing one TLD per line, with support for # comment prefixes. Just add your bespoke internal TLD to the file and it will then pass the host/domain/fqdn validations.

Testing

Continuous Integration is run on this repo with tests for success and failure scenarios:

• unit tests for the custom supporting java library
• integration tests of the top level programs using the libraries for things like option parsing
• functional tests for the top level programs using Docker containers

To trigger all tests run:

make test

which will start with the underlying libraries, then move on to top level integration tests and functional tests using docker containers if docker is available.

Kafka 0.8 support - Alternative Perl & Python Kafka API Nagios Plugins

The Advanced Nagios Plugins Collection has both Perl and Python predecessors to this program which work with Kafka 0.8+. The main differenitator with this Scala version is that it uses the new native 0.9+ Java API which has Kerberos support (the dynamic language versions were built on libraries for Kafka 0.8).

git.io/nagios-plugin-kafka

More Core Repos

Knowledge

DevOps Code

Containerization

CI/CD

DBA - SQL

DevOps Reloaded

Templates

Misc

The rest of my original source repos are here.

Pre-built Docker images are available on my DockerHub.