IPv4/IPv6 interop issue when connecting to local Kafka

[marlier]

(Note, this is now working for me, wanted to leave this here in case it's something you expect kafka-python to handle. Or for others who search for the problem and find it.)

OS: Mac OS X 10.13.3
Python version: 2.7.14

Kafka is running locally, and kafka-console-consumer and kafka-console-producer both work as expected. Listener config is
listeners=PLAINTEXT://localhost:9092

Attempting to instantiate a producer results in the following:

>>> p = KafkaProducer(bootstrap_servers=['localhost:9092'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/imarlier/.virtualenvs/coal/lib/python2.7/site-packages/kafka/producer/kafka.py", line 347, in __init__
    **self.config)
  File "/Users/imarlier/.virtualenvs/coal/lib/python2.7/site-packages/kafka/client_async.py", line 221, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
  File "/Users/imarlier/.virtualenvs/coal/lib/python2.7/site-packages/kafka/client_async.py", line 826, in check_version
    raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable

Increasing log level and attempting to connect gives this, which indicates that the producer is attempting to use IPv6 to connect:

>>> p = KafkaProducer(bootstrap_servers=['localhost:9092'])
DEBUG:kafka.producer.kafka:Starting the Kafka producer
DEBUG:kafka.metrics.metrics:Added sensor with name connections-closed
DEBUG:kafka.metrics.metrics:Added sensor with name connections-created
DEBUG:kafka.metrics.metrics:Added sensor with name select-time
DEBUG:kafka.metrics.metrics:Added sensor with name io-time
INFO:kafka.client:Bootstrapping cluster metadata from [('localhost', 9092, 0)]
DEBUG:kafka.client:Attempting to bootstrap via node at localhost:9092
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent-received
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name request-latency
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.latency
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap host=localhost/localhost port=9092>: creating new socket
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap host=localhost/::1 port=9092>: setting socket option (6, 1, 1)
INFO:kafka.conn:<BrokerConnection node_id=bootstrap host=localhost/::1 port=9092>: connecting to ::1:9092
ERROR:kafka.conn:Connect attempt to <BrokerConnection node_id=bootstrap host=localhost/::1 port=9092> returned error 61. Disconnecting.
INFO:kafka.conn:<BrokerConnection node_id=bootstrap host=localhost/::1 port=9092>: Closing connection. ConnectionError: 61
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap host=localhost/::1 port=9092>: reconnect backoff 0.051544524358 after 1 failures
ERROR:kafka.client:Unable to bootstrap from [('localhost', 9092, 0)]
INFO:kafka.producer.kafka:Kafka producer closed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/imarlier/.virtualenvs/coal/lib/python2.7/site-packages/kafka/producer/kafka.py", line 347, in __init__
    **self.config)
  File "/Users/imarlier/.virtualenvs/coal/lib/python2.7/site-packages/kafka/client_async.py", line 221, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
  File "/Users/imarlier/.virtualenvs/coal/lib/python2.7/site-packages/kafka/client_async.py", line 826, in check_version
    raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable

Changing the bootstrap server to use 127.0.0.1:9092 instead, the producer connects, but isn't able to establish the broker connection because of the same issue -- Kafka returns 'localhost:9092' as the broker address.

>>> p = KafkaProducer(bootstrap_servers=['127.0.0.1:9092'])
DEBUG:kafka.producer.kafka:Starting the Kafka producer
DEBUG:kafka.metrics.metrics:Added sensor with name connections-closed
DEBUG:kafka.metrics.metrics:Added sensor with name connections-created
DEBUG:kafka.metrics.metrics:Added sensor with name select-time
DEBUG:kafka.metrics.metrics:Added sensor with name io-time
INFO:kafka.client:Bootstrapping cluster metadata from [('127.0.0.1', 9092, 2)]
DEBUG:kafka.client:Attempting to bootstrap via node at 127.0.0.1:9092
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent-received
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name request-latency
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.latency
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: creating new socket
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: setting socket option (6, 1, 1)
INFO:kafka.conn:<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: connecting to 127.0.0.1:9092
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: established TCP connection
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: Connection complete.
DEBUG:kafka.client:Node bootstrap connected
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092> Request 1: MetadataRequest_v0(topics=[])
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092> Response 1: MetadataResponse_v0(brokers=[(node_id=0, host=u'localhost', port=9092)], topics=[(error_code=0, topic=u'eventlogging_NavigationTiming', partitions=[(error_code=0, partition=0, leader=0, replicas=[0], isr=[0])]), (error_code=0, topic=u'__consumer_offsets', partitions=[(error_code=0, partition=23, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=41, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=32, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=8, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=17, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=44, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=35, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=26, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=11, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=29, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=38, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=47, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=20, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=2, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=5, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=14, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=46, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=49, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=40, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=4, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=13, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=22, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=31, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=16, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=7, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=43, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=25, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=34, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=10, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=37, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=1, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=19, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=28, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=45, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=36, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=27, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=9, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=18, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=21, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=48, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=12, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=3, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=30, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=39, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=15, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=42, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=24, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=33, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=6, leader=0, replicas=[0], isr=[0]), (error_code=0, partition=0, leader=0, replicas=[0], isr=[0])])])
DEBUG:kafka.cluster:Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 2, groups: 0)
INFO:kafka.client:Bootstrap succeeded: found 1 brokers and 2 topics.
INFO:kafka.conn:<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: Closing connection.
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: reconnect backoff 0.0586890909222 after 1 failures
DEBUG:kafka.client:Initiating connection to node 0 at localhost:9092
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.latency
DEBUG:kafka.conn:<BrokerConnection node_id=0 host=localhost/localhost port=9092>: creating new socket
DEBUG:kafka.conn:<BrokerConnection node_id=0 host=localhost/::1 port=9092>: setting socket option (6, 1, 1)
INFO:kafka.conn:<BrokerConnection node_id=0 host=localhost/::1 port=9092>: connecting to ::1:9092
INFO:kafka.producer.kafka:Kafka producer closed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/imarlier/.virtualenvs/coal/lib/python2.7/site-packages/kafka/producer/kafka.py", line 347, in __init__
    **self.config)
  File "/Users/imarlier/.virtualenvs/coal/lib/python2.7/site-packages/kafka/client_async.py", line 221, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
  File "/Users/imarlier/.virtualenvs/coal/lib/python2.7/site-packages/kafka/client_async.py", line 826, in check_version
    raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable

Things started working as expected after changing the Kafka config to listeners=PLAINTEXT://127.0.0.1:9092

[dpkp]

Have you tried v1.4.2 ? This was fixed via #1411

[marlier]

You're right. I'm using 1.3.5. Not clear why, since I installed via pip on April 5th, but it does explain this. Thanks.