[IMPROVED] Memory based streams and NRG behavior during server restarts #5506
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With memory based wals we can not use snapshots on restarts, but we do use them while they are running. However if a server becomes a leader with no snapshot it will be forced to stepdown when asked to catchup a follower. So we now inherit a leaders snapshot. Also when we tried to truncate on a mismatch, we needed to truncate the previous index, not current. When we fail due to the previous entry being compacted away, we would reset. We now reset the wal to the prior index and use the truncate term and index. Lastly if we receive a heartbeat with correct index but newer term just inherit. Signed-off-by: Derek Collison <[email protected]>
Signed-off-by: Derek Collison <[email protected]>
Signed-off-by: Derek Collison <[email protected]>
These are mostly for memory based streams and cosnumers that need to rebuild complete state on a server restart. For stream health checks for replicated streams make sure that the monitor routine is running. When waiting on consumer assignments at the beginning of the stream monitor, make sure the consumer monitor is running as well if replicated. On a consumer snapshot, register pre-acks as needed. On stream checkInterestState reset an empty stream to the low ack floor from all consumers. Signed-off-by: Derek Collison <[email protected]>
philpennock
reviewed
Jun 10, 2024
Signed-off-by: Derek Collison <[email protected]>
…ueue based streams. Signed-off-by: Derek Collison <[email protected]>
derekcollison
force-pushed
the
mem-wq-restarts
branch
from
08661af
to
6e4ff50
Compare
…ddress in separate CL or PR Signed-off-by: Derek Collison <[email protected]>
wallyqs
pushed a commit
that referenced
this pull request
Jun 10, 2024
…ts (#5506) Improvements to catchups and health checks. Improvements to handling snapshots for memory based wals. With memory based wals we can not use snapshots on restarts, but we do use them while they are running. However if a server becomes a leader with no snapshot it will be forced to stepdown when asked to catchup a follower. So we now inherit a leaders snapshot. Also when we tried to truncate on a mismatch, we needed to truncate the previous index, not current. When we fail due to the previous entry being compacted away, we would reset. We now reset the wal to the prior index and use the truncate term and index. Lastly if we receive a heartbeat with correct index but newer term just inherit. For stream health checks for replicated streams make sure that the monitor routine is running. When waiting on consumer assignments at the beginning of the stream monitor, make sure the consumer monitor is running as well if replicated. On a consumer snapshot, register pre-acks as needed. On stream checkInterestState reset an empty stream to the low ack floor from all consumers. Last fix consistency bug with memstore when skipping msgs on empty stream to ensure first == last + 1. Signed-off-by: Derek Collison <[email protected]> --------- Signed-off-by: Derek Collison <[email protected]>
This was referenced Aug 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Improvements to catchups and health checks.
Improvements to handling snapshots for memory based wals.
With memory based wals we can not use snapshots on restarts, but we do use them while they are running.
However if a server becomes a leader with no snapshot it will be forced to stepdown when asked to catchup a follower. So we now inherit a leaders snapshot.
Also when we tried to truncate on a mismatch, we needed to truncate the previous index, not current.
When we fail due to the previous entry being compacted away, we would reset. We now reset the wal to the prior index and use the truncate term and index.
Lastly if we receive a heartbeat with correct index but newer term just inherit.
For stream health checks for replicated streams make sure that the monitor routine is running.
When waiting on consumer assignments at the beginning of the stream monitor, make sure the consumer monitor is running as well if replicated.
On a consumer snapshot, register pre-acks as needed.
On stream checkInterestState reset an empty stream to the low ack floor from all consumers.
Last fix consistency bug with memstore when skipping msgs on empty stream to ensure first == last + 1.
Signed-off-by: Derek Collison [email protected]