Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][broker] PIP-379: Snapshot hash range assignments only in AUTO_SPLIT ordered mode #23423

Merged
merged 1 commit into from
Oct 8, 2024

Conversation

lhotari
Copy link
Member

@lhotari lhotari commented Oct 8, 2024

Motivation

The Key_Shared PIP-379 implementation performs snapshots of hash range assignments after adding or removing a consumer in order to detect the existing consumers and hash ranges that changed as a result. This result is only used in Key_Shared AUTO_SPLIT ordered mode. Therefore, it's an unnecessary overhead for all other use cases. Any unnecessary work that can be avoided is a useful optimization, although the performance of the current snapshotting isn't a concern based on the microbenchmark and optimizations performed in the initial PR. (comment, simple microbenchmark).

Modifications

  • modify StickyKeyConsumerSelector interface:
    • CompletableFuture<ImpactedConsumersResult> addConsumer(Consumer consumer) -> CompletableFuture<Optional<ImpactedConsumersResult>> addConsumer(Consumer consumer)
    • ImpactedConsumersResult removeConsumer(Consumer consumer) -> Optional<ImpactedConsumersResult> removeConsumer(Consumer consumer)
  • adjust ConsistentHashingStickyKeyConsumerSelector and HashRangeAutoSplitStickyKeyConsumerSelector to this change
  • drop snapshotting in addConsumer and removeConsumer for HashRangeExclusiveStickyKeyConsumerSelector since it's not needed.
  • adjust PersistentStickyKeyDispatcherMultipleConsumers to configure ConsistentHashingStickyKeyConsumerSelector and HashRangeAutoSplitStickyKeyConsumerSelector classes to use snapshotting when needed (Key_Shared AUTO_SPLIT ordered mode) and adapt to interface changes.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@lhotari lhotari merged commit 84b834f into apache:master Oct 8, 2024
57 of 60 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/broker doc-not-needed Your PR changes do not impact docs ready-to-test
Development

Successfully merging this pull request may close these issues.

3 participants