[improve][broker] PIP-379: Snapshot hash range assignments only in AUTO_SPLIT ordered mode #23423
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The Key_Shared PIP-379 implementation performs snapshots of hash range assignments after adding or removing a consumer in order to detect the existing consumers and hash ranges that changed as a result. This result is only used in Key_Shared AUTO_SPLIT ordered mode. Therefore, it's an unnecessary overhead for all other use cases. Any unnecessary work that can be avoided is a useful optimization, although the performance of the current snapshotting isn't a concern based on the microbenchmark and optimizations performed in the initial PR. (comment, simple microbenchmark).
Modifications
StickyKeyConsumerSelector
interface:CompletableFuture<ImpactedConsumersResult> addConsumer(Consumer consumer)
->CompletableFuture<Optional<ImpactedConsumersResult>> addConsumer(Consumer consumer)
ImpactedConsumersResult removeConsumer(Consumer consumer)
->Optional<ImpactedConsumersResult> removeConsumer(Consumer consumer)
ConsistentHashingStickyKeyConsumerSelector
andHashRangeAutoSplitStickyKeyConsumerSelector
to this changeaddConsumer
andremoveConsumer
forHashRangeExclusiveStickyKeyConsumerSelector
since it's not needed.PersistentStickyKeyDispatcherMultipleConsumers
to configureConsistentHashingStickyKeyConsumerSelector
andHashRangeAutoSplitStickyKeyConsumerSelector
classes to use snapshotting when needed (Key_Shared AUTO_SPLIT ordered mode) and adapt to interface changes.Documentation
doc
doc-required
doc-not-needed
doc-complete