Broker offset store method does not work (implement ConsumerMetadataRequest)

[npotereyko]

I have consumer which works with with default store method ("file").
But when I set topic conf "offset.store.method" to "broker" everything stops working - I keep getting messages with "Broker: Unknown topic or partition" error set, and offset is -1.

I do have Kafka 2.9.2-0.8.1.1 installed, and also global config "group.id" set to "group1". Not sure what I am missing.
I am using librdkafka-0.8.5 release.

[edenhill]

While I dont recognize this error could you please try to reproduce this on the latest master?

[edenhill]

The behaviour on the broker has changed since this was implemented in librdkafka,
I'll make sure to update the library to support the new functionality.

[edenhill]

Seems like the OffsetCommit API semantics changed in 0.8.1.1, and from what I understand it will not be properly implemented until Kafka 0.9, so there isn't much I can do at this point.

[hendrik-schumacher]

Are you sure about that? After all the "new offset management" of version 0.8.2 is advertised all over the place. From what I understand the internal offset storage still needs testing and documentation but the offset management api itself should work. There is a sample client patch attached to KAFKA-1784.

Nevertheless I think you have more insights than me so: if the OffsetCommit API is not usable anymore do we have to rely on file based storage when using librdkafka? That would be somehow a step backwards from 0.8.1.

Thanks for your help.

[edenhill]

You are probably right, I will take a closer look. My previous statement was based on sketchy functionality in a 0.8.2 beta.

[DEvil0000]

Anything new on this one?

[edenhill]

Will look into it during this week, thanks for your patience.

[DEvil0000]

Thanks

[wizzat]

I've been having some trouble with creating consumers that support broker metadata offset requests. For me, messages just never come through at all. Is the current status that rdkafka doesn't support broker metadata requests past version 0.8.1.0?

[skandragon]

It would be good to have this working, but if it's broken, at least the docs should say don't use it, rather than recommending it as the preferred method.

[lzshlzsh]

Any good news to this problem?

[edenhill]

Dup #227

[edenhill]

Sorry for the slow response to this issue, it will soon be reprioritized and fixed.

[matsbror]

Was this ever fixed for 0.8?
I am trying to use it, but the consumer callback is never called.

[edenhill]

@matsbror Yes, this works with broker 0.8.2 and later. Just make sure to configure group.id and calling consume_start() with offset set to RD_KAFKA_OFFSET_STORED

Let me know how it goes.

example:
examples/rdkafka_example -b yourbroker -t yourtopic -p 0 -C -X group.id=yourgroup -o stored -X topic.auto.offsereset=smallest

[matsbror]

Thanks! I got it working now. I had set topic store method to "file" which is why it didn't work.
Mats

Date: Tue, 22 Mar 2016 02:08:07 -0700
From: [email protected]
To: [email protected]
CC: [email protected]
Subject: Re: [librdkafka] Broker offset store method does not work (implement ConsumerMetadataRequest) (#192)

@matsbror Yes, this works with broker 0.8.2 and later. Just make sure to configure group.id and calling consume_start() with offset set to RD_KAFKA_OFFSET_STORED

Let me know how it goes.

example:

examples/rdkafka_example -b yourbroker -t yourtopic -p 0 -C -X group.id=yourgroup -o stored -X topic.auto.offsereset=smallest

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub