Support partitioned topics in the Reader interface #7265
Assignees
Labels
area/client
help wanted
lifecycle/stale
type/feature
The PR added a new feature or issue requested a new feature
Comments
yoniiny
added
the
type/feature
|
Please assign it to me |
|
Any progress on this? It's a massive blocker for us. |
codelipenghui
pushed a commit
that referenced
this issue
Nov 17, 2020
Fixes #3643 #7265 ### Motivation Support partitioned topics in the Reader ### Modifications class relationship： PulsarClientImpl -> MultiTopicsReaderImpl -> MultiTopicsConsumerImpl -> ConsumerImpl PulsarClientImpl support build MultiTopicsReader MultiTopicsReader wraps one MultiTopicsConsumerImpl MultiTopicsConsumerImpl contains multiple consumerImpl Make MultiTopicsConsumerImpl support seek by messageId seek by time: All consumerImpl seek by time, Reader can get all the messages returned by each partition seek by message： 1）When seek by latest/earliest,all partitions seek by latest/earliest,Reader can get all the messages returned by each partition 2）When the messageId contains explicit partition information, it will only receive messages from this partition, and will not receive messages from other partitions.To avoid other partitions returning messages in inclusive mode, I make other consumer seek to latest and add a ignoredConsumersSet to ignore messages from other partitions .
flowchartsman
pushed a commit
to flowchartsman/pulsar
that referenced
this issue
Nov 17, 2020
Fixes apache#3643 apache#7265 ### Motivation Support partitioned topics in the Reader ### Modifications class relationship： PulsarClientImpl -> MultiTopicsReaderImpl -> MultiTopicsConsumerImpl -> ConsumerImpl PulsarClientImpl support build MultiTopicsReader MultiTopicsReader wraps one MultiTopicsConsumerImpl MultiTopicsConsumerImpl contains multiple consumerImpl Make MultiTopicsConsumerImpl support seek by messageId seek by time: All consumerImpl seek by time, Reader can get all the messages returned by each partition seek by message： 1）When seek by latest/earliest,all partitions seek by latest/earliest,Reader can get all the messages returned by each partition 2）When the messageId contains explicit partition information, it will only receive messages from this partition, and will not receive messages from other partitions.To avoid other partitions returning messages in inclusive mode, I make other consumer seek to latest and add a ignoredConsumersSet to ignore messages from other partitions .
|
#7518 has fixed the issue, so close for now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
Working with large volumes of data requires partitioned topics. Ensuring exactly-once using idempotent operations requires using the Reader interface, but it doesn't support partitioned topics.
Describe the solution you'd like
The Reader interface should support partitioned topics
Describe alternatives you've considered
The issue #3643 would be a workaround that would allow this to work by creating and destroying Consumers. This is quite clunky though.
The text was updated successfully, but these errors were encountered: