consumer keep re-consuming last message from each partition

[npotereyko]

Not sure whether this is a feature or bug, but every time when I start my consumers they re-consume last event from each partition they previously consumed in previous run.
Offsets are stored in file, and after consumer finishes - I do see that offset of last consumed message is stored in ".offset" file.
But when consumer starts again, why would I consume same message again?? Shouldn't it resume from next one?
I am using librdkafka-0.8.5 release.

[edenhill]

I think this might be the same as issue #164 .
Could you update to the latest librdkafka master and see if the problem persists?

[DEvil0000]

sounds like dup #164

[hendrik-schumacher]

I still see this issue in a perl script using librdkafka via XS bindings (even with the latest master). If I run consume with callback and immediately call consume stop afterwards, it seems not to have the eof offset yet (I printed it and it is 0). If I put in a sleep in the consume callback it gets the eof offset and it works fine (so it seems to be some timing issue). I can try to put something together that can reproduce this but it is not that easy to break this down with all the XS binding stuff.

[edenhill]

Thanks for the details Hendrik, should be enough to let me reproduce it.

Are you planning on open sourcing your Perl bindings, btw?

[hendrik-schumacher]

I would really like to open source the Perl bindings but I have to wrap them up first. There is still a lot to do before I can release something.

Do you have any update on the re-consumption-issue? Thanks for looking into this.

[edenhill]

@hendrik-schumacher Can you give it a go now?
This should fix a number of offset problems.

[hendrik-schumacher]

I tried it and it works now. Nice work.

I will start doing bigger production-like tests soon and get back if anything looks strange.

Thanks a lot.

[edenhill]

Great!

I have a request from the homebrew people to release a new librdkafka version today and it'd be great if your additional tests were done by then, any chance of that?

[hendrik-schumacher]

I got too much else to do today so I will only be able to run this next week. Sorry. But the package looks good to me now.

[edenhill]

Fair enough, thanks anyway!