Improve batch message acking by removing batch message tracker #1424
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
merlimat
reviewed
Mar 23, 2018
| } | ||
|
|
||
| // bitset shared across messages in the same batch. | ||
| private final BitSet bitSet; |
There was a problem hiding this comment.
Since we have many of these objects, we could inherit from BitSet to avoid an extra object.
There was a problem hiding this comment.
hmm then I have to change how BatchMessageAckerDisabled and such. I think this change here removes the hashmap, which already remove bunch of allocations per batch message. It might be better to hide bitset behind the acker and make batch message acker a recycle object, which we can reuse those objects.
There was a problem hiding this comment.
Sure, we can do that later
| if (batchMessageAckTracker.isEmpty()) { | ||
| return; | ||
| } | ||
| MessageIdImpl lowerKey = batchMessageAckTracker.lowerKey(message); |
There was a problem hiding this comment.
I don't see how this logic to remove all entries from previous batches (when doing cumulative ack) has been replaced.
|
retest this please |
|
@sijie can you merge with master? It already has the flaky test fixed. |
|
merged latest master |
|
@sijie I think there might be a legitimate test failure: https://builds.apache.org/job/pulsar_precommit_java8/55/testReport/junit/org.apache.pulsar.client.api/SimpleProducerConsumerStatTest/testMessageListener/ |
jai1
approved these changes
Mar 28, 2018
|
@merlimat will take a look |
jai1
approved these changes
Mar 28, 2018
2 tasks
BewareMyPower
added a commit
to BewareMyPower/pulsar-client-cpp
that referenced
this pull request
Nov 30, 2022
Fixes apache#130 ### Motivation It's a catchup for apache/pulsar#1424 ### Modifications Migrate `BitSet` implementation from JDK. Though we have `vector<bool>` or `boost::dynamic_bitset`, to support batch index ACK in future, it's better to have the same method to convert a bit set to the underlying long array. Add `BatchedMessageIdImpl` to maintain a `BatchMessageAcker`, which is shared by messages in the same batch. The acker is responsible to record which messages are acknowledged. Only if all messages in the batch are acknowledged will the message id be acknowledged. For cumulative ack, use the number of removed batches to update the stats. (TODO: record the batch size in the unacked message tracker in future).
BewareMyPower
added a commit
to BewareMyPower/pulsar-client-cpp
that referenced
this pull request
Dec 1, 2022
Fixes apache#130 ### Motivation It's a catchup for apache/pulsar#1424 ### Modifications Migrate `BitSet` implementation from JDK. Though we have `vector<bool>` or `boost::dynamic_bitset`, to support batch index ACK in future, it's better to have the same method to convert a bit set to the underlying long array. Add `BatchedMessageIdImpl` to maintain a `BatchMessageAcker`, which is shared by messages in the same batch. The acker is responsible to record which messages are acknowledged. Only if all messages in the batch are acknowledged will the message id be acknowledged.
BewareMyPower
added a commit
to BewareMyPower/pulsar-client-cpp
that referenced
this pull request
Dec 2, 2022
Fixes apache#130 ### Motivation It's a catchup for apache/pulsar#1424 ### Modifications Migrate `BitSet` implementation from JDK. Though we have `vector<bool>` or `boost::dynamic_bitset`, to support batch index ACK in future, it's better to have the same method to convert a bit set to the underlying long array. Add `BatchedMessageIdImpl` to maintain a `BatchMessageAcker`, which is shared by messages in the same batch. The acker is responsible to record which messages are acknowledged. Only if all messages in the batch are acknowledged will the message id be acknowledged. The stats for individual ACKs only updates after the whole batch are acknowledged. Before this PR, each time the single message is acknowledged, the stats increase by one.
BewareMyPower
added a commit
to apache/pulsar-client-cpp
that referenced
this pull request
Dec 8, 2022
* Add BatchedMessageIdImpl to acknowledge batched messages Fixes #130 ### Motivation It's a catchup for apache/pulsar#1424 ### Modifications Migrate `BitSet` implementation from JDK. Though we have `vector<bool>` or `boost::dynamic_bitset`, to support batch index ACK in future, it's better to have the same method to convert a bit set to the underlying long array. Add `BatchedMessageIdImpl` to maintain a `BatchMessageAcker`, which is shared by messages in the same batch. The acker is responsible to record which messages are acknowledged. Only if all messages in the batch are acknowledged will the message id be acknowledged. The stats for individual ACKs only updates after the whole batch are acknowledged. Before this PR, each time the single message is acknowledged, the stats increase by one.
2 tasks
2 tasks
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
client is maintaining a batch message tracker to track batch messages for acknowledge. if the application is using
Failoversubscription and doesn't ack the messages at all, the message id objects will be accumulating in the client.Modifications
Move the bitset as part of BatchMessageIdImpl and remove batch message tracker