Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consumer group reassignment of offset not working #59

Open
SrikarSaggurthi opened this issue Jul 27, 2020 · 1 comment
Open

Consumer group reassignment of offset not working #59

SrikarSaggurthi opened this issue Jul 27, 2020 · 1 comment
Labels

Comments

@SrikarSaggurthi
Copy link

Describe the bug
For an existing consumer group with stored offsets, resetting the offset to the latest is not working, it still reads from the stored offset.

To Reproduce

  1. First, have to set up a consumer group (say test) and run for a few messages and commit those offsets. .kfk.Sub[client;topicName;(enlist 6h$1)!(enlist .kfk.OFFSET.STORED)];

  2. Now restart the process and assign a specific partition and latest offset with same group test .kfk.Sub[client;topicName;(enlist 6h$1)!(enlist .kfk.OFFSET.END)];`

Expected behavior
It should only start reading if there are new messages in `topicName, however it starts reading from the last committed offset.

Desktop (please complete the following information):

  • OS: linux 2.6
  • KDB+ banner information KDB+ 3.6 2019.09.19
  • .kfk.Version : 16842992i

Additional context
relevant config parmater "auto.offset.reset:latest"

@SrikarSaggurthi SrikarSaggurthi changed the title consumer group manual assignment of offset not working Consumer group reassignment of offset not working Jul 27, 2020
@cmccarthy1 cmccarthy1 added the bug label Aug 2, 2020
@mshimizu-kx
Copy link
Contributor

The partition is ignored at subscribe as described here.

Maybe you can assign a new offset for the consumer with .kfk.assignOffsets or .kafka.assignNewOffsetsToTopicPartition.

.kafka.subscribe[consumer;topic1];
.kafka.subscribe[consumer;topic2];
.kafka.assignNewOffsetsToTopicPartition[consumer; ; (1#0i)!1#.kafka.OFFSET_END] each (topic1; topic2);

You can check the behavior with the test_consumer.q example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants