[improve][ml] Filter out deleted entries before read entries from ledger.

[dao-jun]

PIP: I'm not sure if the PR needs a PIP

Motivation

In #19035 we introduced skipCondition before read entries from ledger.

It filters out delay-messages for the purpose of optimizing performance: to avoid reading unnecessary entries to save memory usage, network usage and reduce latency of reading entries.

As the same purpose, this PR wants to filter-out deleted entries before reading entries from ledger to optimize performance.

This PR won't change the semantics of readEntries , it just made some enhancements.

It may brings huge performance optimization in the individualDelete mode, especially when bookie client metadata loss, bookie auto-recover disabled and some bookie nodes down.

Modifications

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: dao-jun#4

[github-actions]

@dao-jun Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[dao-jun]

/pulsarbot run-failure-checks

[codelipenghui]

It may brings huge performance optimization in the individualDelete mode, especially when bookie client metadata loss, bookie auto-recover disabled and some bookie nodes down.

Could you please explain more about this line? I don't understand why it pertains to bookie client metadata loss, the disabled auto-recovery and bookies unavailability.

[dao-jun]

@codelipenghui @coderzc PTAL

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (22838ea) 36.46% compared to head (276358b) 73.58%.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #21739       +/-   ##
=============================================
+ Coverage     36.46%   73.58%   +37.11%     
- Complexity    12390    32381    +19991     
=============================================
  Files          1725     1861      +136     
  Lines        131701   138596     +6895     
  Branches      14401    15187      +786     
=============================================
+ Hits          48027   101985    +53958     
+ Misses        77254    28700    -48554     
- Partials       6420     7911     +1491     

Flag	Coverage Δ
inttests	24.14% <0.00%> (+0.02%)	⬆️
systests	23.71% <0.00%> (+0.04%)	⬆️
unittests	72.86% <100.00%> (+40.81%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...che/bookkeeper/mledger/impl/ManagedCursorImpl.java	79.22% <100.00%> (+30.88%)	⬆️
...che/bookkeeper/mledger/impl/ManagedLedgerImpl.java	80.53% <ø> (+32.69%)	⬆️
...he/bookkeeper/mledger/impl/ReadOnlyCursorImpl.java	95.00% <100.00%> (+95.00%)	⬆️

... and 1435 files with indirect coverage changes

[coderzc]

approve

[codelipenghui]

I have added the cherry-pick labels since this improvement fixed the issue which can impact the bookies if the cursor rewind to the earlier position (the mark delete position can't move forward due to individual acks).