[feat][broker] PIP-180 Part VI: Add ShadowManagedLedgerImpl #12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
14 tasks
…he#18252) * [fix][broker] Fix if dynamicConfig item in ZK do not exist in broker cause NPE (apache#17705) * [fix][Authorization] Modify authorization to update topic's properties Co-authored-by: Lei Zhiyuan <[email protected]>
…pache#17847) Master Issue: apache#16913 ### Motivation Implement an abortedTxnProcessor to handle the storage of the aborted transaction ID. ### Modifications The structure overview:  The main idea is to move the logic of the operation of checking and persistent aborted transaction IDs(take snapshots) and the operation of updating maxReadPosition into the AbortedTxnProcessor. And the AbortedTxnProcessor can be implemented in different designs. **Add `persistentWorker` to handle snapshot persistenting** ： <img width="1003" alt="image" src="https://user-images.githubusercontent.com/55571188/198528131-3cde19bc-2034-4693-a8b1-4d6345e6db36.png"> The first four items below are the corresponding four tasks in the figure. The fifth item is not strictly a task, but a part of the first two tasks. * takeSnapshotSegmentAsync -> writeSnapshotSegmentAsync * These two method is used to persist the snapshot segment. * deleteSnapshotSegment * This method is used to delete the snapshot segment. * updateIndexMetadataForTheLastSnapshot * Using to update index metadata (the latest snapshot). * clearSnapshotSegmentAndIndexes * Delete all segments and then delete the index of this topic. * updateSnapshotIndex * Called by the deleteSnapshotSegment and writeSnapshotSegmentAsync. Do update the index after writing snapshot segment. * Called by recovery as a compensation mechanism for updating the index. ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [x] `doc-not-needed` (Please explain why) ### Matching PR in the forked repository PR in forked repository: liangyepianzhou#7
…with broker property (apache#17803)
…mmandSender` (apache#18250) change to `sendSuccessResponse` and `sendErrorResponse`
… ExecutorProvider.ExtendedThreadFactory
* fix some error * Update site2/docs/functions-overview.md Co-authored-by: tison <[email protected]> Co-authored-by: tison <[email protected]>
…pache#18141) * Move out some already stable test from flaky group. * Assigned tests to some test group explicitly * fix test * Address comment
…r cli (apache#17295) Signed-off-by: Zixuan Liu <[email protected]>
…on topic (apache#18212) * [improve] [client] Add api to get producer/consumer stats for partition topic * introduce partition topic stats interface
…ache#18289) * [fix][test]fix flaky test SimpleProducerConsumerTestStreamingDispatcherTest.testSharedSamePriorityConsumer * remove unnecessary change
…8278) * [cleanup] Direct get lowest PositionImpl from TreeMap change signature from Set<T> to NavigableSet<T> which makes the caller to get lowest PositionImpl more efficient. * change poll to first when call `NavigableSet` * fix check style remove unused import Co-authored-by: wangjinlong <[email protected]>
… and trying to access it (apache#17901)
…he#18310) Fixes apache#18230 ### Motivation `ManagedCursorImpl` has a known problem: when the `cursor.close` and the `cursor.switchLedger` are concurrent executing, a bad version exception is thrown. see: https://github.com/apache/pulsar/blob/0c3c175eb132d4ef0fb3a12842127dcb4fa1933d/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java#L2595-L2619 In this test, it hit this problem: - `MlPendingAckStore.clearUselessLogData` will trigger `cursor.switchLedger` if it is the first execute `mard delete`. - `MlPendingAckStore.close` will trigger `cursor.close` ### Modifications if `MlPendingAckStore.close` fail, just retry, it will be ok
…ile have active consumers (apache#18283)
reason: https://lists.apache.org/thread/jcrq9q0k6kh9rvb71dwb9s3mvo5c5dk5 - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
reason: https://lists.apache.org/thread/jcrq9q0k6kh9rvb71dwb9s3mvo5c5dk5 branch-2.9 has reset to commit `5ed247de3a5e0ff67f181b4805fd6140d8174994`, so close force-push - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
…pache#18603) This PR is a supplement to apache#18369. - `AbstractTopic.isSameAddressProducersExceeded()` - `AbstractBaseDispatcher.isConsumersExceededOnSubscription()`
… to cache the string hashing (apache#18566)
Signed-off-by: tison <[email protected]>
…erSingleActiveConsumer (apache#18648)
…ce usage(>100%) in load balancer (apache#18645)
- finish processSourceManagedLedgerInfo - finish internalAsyncAddEntry - ShadowTopicTest pass - ShadowReplicatorTest pass Add test cases
Jason918
force-pushed
the
pip-180-dev
branch
from
69b8222 to
536f10a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is for running tests for upstream PR apache#18265.