Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][broker][PIP-195] Cut off snapshot segment according to maxIndexesPerBucketSnapshotSegment #19706

Merged
merged 3 commits into from
Mar 14, 2023

Conversation

coderzc
Copy link
Member

@coderzc coderzc commented Mar 3, 2023

PIP: #16763

Motivation

currently, the tracker only cut off snapshot segments according to the time range, to prevent single snapshot segment oversize we should add the max number limitation for snapshot segment.

Modifications

  • Cut off snapshot segment according to config delayedDeliveryMaxIndexesPerBucketSnapshotSegment.
  • Remove unimplemented configuration delayedDeliverySharedIndexEnabled
  • Improve variable types

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@coderzc coderzc self-assigned this Mar 3, 2023
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 3, 2023
@coderzc coderzc added the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Mar 3, 2023
@coderzc coderzc added this to the 3.0.0 milestone Mar 3, 2023
@coderzc coderzc force-pushed the bucket_delayed_max_indexes_per_segment branch from 1adaaa4 to d917561 Compare March 3, 2023 12:08
@codelipenghui
Copy link
Contributor

/pulsarbot run-failure-checks

@mattisonchao
Copy link
Member

Please don't forget to add configuration to broker.conf and standalone.conf

@coderzc coderzc merged commit da3cab5 into apache:master Mar 14, 2023
@coderzc coderzc deleted the bucket_delayed_max_indexes_per_segment branch March 14, 2023 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/broker doc-not-needed Your PR changes do not impact docs ready-to-test type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants