(local) consumer groups: not working

[DEvil0000]

i know the broker based consumer group is not implemented yet #152.
but it would be possible to handle it at least local in process or on local for the machine (see offset files #199 ).

i was trying to use two consumers in the same group (group name "group 1") in the same process but they consumed both the same message what was kind of unexpected.
using topicConfig->set("group.id", stringGroupName, errorStr)

[edenhill]

This is not a trivial task since it requires consumer group correlation, which is what will be implemented with #152 anyway :)

[DEvil0000]

simple quick fix:
lock offset file on consume..
check offset for "my" group..
write offset to file immediate..
unlock file..

[edenhill]

Consumer groups splits consumers up on partitions, not specific messages, i.e.: consumer 1 reads partition 2, consumer 2 reads partition 3, consumer 3 reads partition 1.

This should be fairly easy to implement on top of librdkafka with some IPC calls between processes.

[DEvil0000]

hehe i simplified the problem (maybe too much).
i only use 1 partition so i don't care about distribution of consumers on partitions (load balance).
the idea was to use a consumer group for fault tolerance.

i have to make sure a message is only processed once but i don't care if it gets processed by process a or b (on machine a or b). this is the way i would like to use consumer groups.
and as long as broker offsets and broker managed consumer groups are not available it would be ok if it would work local with multi processes. (like the java thingi started from the shell)

i was hoping this would be allready in rdkafka or at least a easy thing. if this is not the case (and it looks like) i have to rethink my approach :(

edit:
all my real needs seam to come down to #4. are you working on this? when may i expect this?

[edenhill]

#4 probably wont be implemented since Kafka is moving away from ZooKeeper reliance.
librdkafka will support broked based consumer groups though, tracking issue #152

[DEvil0000]

the problem is i might need this feature until beginning of april ;) i think i will write a wrapping with zookeeper like in the example