Kafka client doesn’t resolve localhost on a Mac

[Glutexo]

Currently, our development docker-compose file configures Kafka to use only a single advertised listener. This listener is used both for the internal communication inside Docker and for the communication with the outside world – our app. The only way to use this listener is to create a hosts entry for kafka pointing to 127.0.0.1 to mimic the internal Docker environment.

…
services:
  …
  kafka:
    …
    environment:
        …
        - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092
        …

127.0.0.1 kafka

This is a mere workaround and there is a better way to configure the advertised listeners: to have one for the internal communication and one for the external. Here is a nice, understandable tl;dr for that.

…
services:
  …
  kafka:
    …
    environment:
        …
        - KAFKA_LISTENERS=INTERNAL://kafka:9092,EXTERNAL://kafka:29092
        - KAFKA_ADVERTISED_LISTENERS=INTERNAL://kafka:9092,EXTERNAL://localhost:29092
        - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
        - KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
        …

Although this is the “correct” settings, there is still a problem with that. If Kafka is configured to use localhost as a hostname for external connection, the Inventory app cannot connect, failing with a NoBrokersAvailable error. At least on a Mac.

kafka.errors.NoBrokersAvailable: NoBrokersAvailable

The reason is that localhost is a special hostname that resolves to several IP addresses. The version of kafka-python we use (1.3.5) doesn’t try all the addresses and fails fast if it cannot connect using the first in order.

This is a known issue and has been fixed already. However, the Kafka client package is listed under two different names on PyPI:

• kafka, which we are using, does not receive updates any more and the last release 1.3.5 has the aforementioned bug.
• kafka-python with the latest version being 1.4.6 where the bug is already fixed

When the newer version is used, it is possible to connect to Kafka using the localhost advertised listener.

Steps to reproduce:

1. Deploy the fixed dev.yml docker-compose bundle. docker-compose -f dev.yml up
2. Make sure you have the 29092 port properly forwarded to Docker and possibly Docker-machine. Following Python code can help:

import socket
print(socket.getaddrinfo("localhost", 29092))'

3. Make sure you don’t have a kafka entry in /etc/hosts.

$ ping kafka                                                                    
ping: cannot resolve kafka: Unknown host

4. Run the application with proper Kafka configuration.

$ KAFKA_TOPIC=platform.system-profile KAFKA_EVENT_TOPIC=platform.inventory.events KAFKA_GROUP=inventory KAFKA_BOOTSTRAP_SERVERS=localhost:29092 pipenv run python run_gunicorn.py

5. See that it crashes, because of not being able to connect to Kafka.

kafka.errors.NoBrokersAvailable: NoBrokersAvailable

6. Upgrade the kafka-python package to the fixed version.

$ pipenv run pip uninstall kafka
$ pipenv run pip install kafka-python

7. Restart the application and see that it connects to Kafka without any issues.

If the steps above don’t work for you, please drop a comment.