KeyError Creating SimpleProducer

[herriojr]

This is an issue I've been experiencing running Master branch. Basically, the first time I access an uncreated topic, I get a KeyError, whereas the second time, I can access it just fine.

import kafka
client = kafka.client.KafkaClient(address, port)
producer = kafka.producer.SimpleProducer(client, "test") # Fails first time called, succeeds thereafter

Traceback (most recent call last):
File "/home/vagrant/.pycharm_helpers/pydev/pydevd.py", line 1532, in
debugger.run(setup['file'], None, None)
File "/home/vagrant/.pycharm_helpers/pydev/pydevd.py", line 1143, in run
pydev_imports.execfile(file, globals, locals) #execute the script
File "/projects/scratch/init.py", line 27, in
main()
File "/home/vagrant/.virtualenvs/app/local/lib/python2.7/site-packages/kafka/producer.py", line 191, in init
self.next_partition = cycle(client.topic_partitions[topic])
KeyError: 'test'

[rdiomar]

Not sure why, but when it's a new topic kafka doesn't return the partition info to the first metadata request. As a workaround, you can do what ensure_topic_creation() in test_integration.py does before creating your producer. We should fix this though.

[dpkp]

Latest master (after merging #111 pull request) changes behavior so the Producer init no longer takes a topic. But this error could still show up on first message send to a new topic. Can you test?

[dpkp]

My testing suggests that when topic auto-creation is enabled on the kafka server (auto.create.topics.enable=True), it will create the topic but return a LeaderNotAvailable error while it does the initial setup asynchronously. this means that the message produce has to wait until the topic is fully setup w/ a broker leader. kafka-python should probably check for this error and optionally retry (w/ resyncing leader metadata?)

[hmahmood]

Any updates on this?

[frgtn]

As of 8811298 this still breaks with KeyError:
Traceback (most recent call last):

  File "test_113.py", line 21, in <module>
    'The message')
  File "<some_prefix>/lib/python2.7/site-packages/kafka/producer.py", line 204, in send_messages
    partition = self._next_partition(topic)
  File "<_some_prefix>/lib/python2.7/site-packages/kafka/producer.py", line 200, in _next_partition
    self.partition_cycles[topic] = cycle(self.client.topic_partitions[topic])
KeyError: 'test_2'

[dpkp]

merged #109 -- please retry w/ master trunk

[imduffy15]

Still seeing this issue on master.

Any updates?

[wizzat]

I can confirm the issue happening, and I've looked into it enough to know why. Right now I am working on beefing up the test suite.

On a scale of 1-10, how much of a blocker is this for you?

-Mark

On Apr 9, 2014, at 3:23, Ian Duffy [email protected] wrote:

Still seeing this issue on master.

Any updates?

—
Reply to this email directly or view it on GitHub.

[imduffy15]

I'm working around it for the moment by creating the topics manually.

Lets go with 4 - Annoying.
On Apr 9, 2014 5:53 PM, "Mark Roberts" [email protected] wrote:

I can confirm the issue happening, and I've looked into it enough to know
why. Right now I am working on beefing up the test suite.

On a scale of 1-10, how much of a blocker is this for you?

-Mark

On Apr 9, 2014, at 3:23, Ian Duffy [email protected] wrote:

Still seeing this issue on master.

Any updates?

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/113#issuecomment-39988130
.

[Korijn]

I'm getting exactly this error constantly, not just on the first try. Not sure why yet.

Traceback (most recent call last):
File "C:.........\logging\handlers\kafkahandler.py", line 42, in emit
self.producer.send_messages(self.topic, msg)
File "C:.........\VirtualEnv\lib\site-packages\kafka\producer.py", line 204, in send_messages
partition = self._next_partition(topic)
File "C:.........\VirtualEnv\lib\site-packages\kafka\producer.py", line 200, in _next_partition
self.partition_cycles[topic] = cycle(self.client.topic_partitions[topic])
KeyError: '.........'

[wizzat]

Do you have a topic without ISR or without a leader? Can you print
kafka-list-topics output?

-Mark

On Tue, May 20, 2014 at 4:22 AM, Korijn van Golen
[email protected]:

I'm getting exactly this error constantly, not just on the first try. Not
sure why yet.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/113#issuecomment-43613360
.

[Korijn]

I feel silly. It was a configuration issue. The kafka-server-start.sh and zookeeper-server-start.sh scripts that come with the packaged install from the ubuntu repo's were not exactly suitable for use with supervisord. After correcting setup all problems were solved. I can post the scripts if you're interested.

[dpkp]

Nonetheless, there's an assumption in the code that we'll always get topic metadata back from the server. We need to handle this more gracefully.

https://github.com/mumrah/kafka-python/blob/master/kafka/producer.py#L219

[Korijn]

Here's an idea: make it possible to work with kafka-python without metadata through a configuration option (or something similar). In the case of missing metadata, raise an exception that indicates that metadata can't be retrieved, and mention in the exception message that you can configure kafka-python not to use metadata. That would have pointed me in the proper direction immediately (figuring out why metadata can't be retrieved), instead of aimlessly searching for a workaround to some issue with kafka-python. Just a though :)

Thanks for the help guys.

[dpkp]

Metadata is necessary to know which broker the producer must connect to for a particular topic-partition. kafka-python doesn't currently expose a super low level producer that requires users to manage leadership discovery / topic metadata. But even if it did, you'd still be forced to deal with metadata. It's a pretty core part of the kafka design.

Nonetheless, this is clearly a bug in how kafka-python handles partition cycles when a topic does not exist (or at least the broker has no metadata available).

[Korijn]

I guess you're right about that.