Support opening an in-use rocksdb as secondary #10209
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ryoqun
commented
May 24, 2020
ryoqun
commented
May 24, 2020
ledger-tool/src/main.rs
Outdated
| @@ -873,7 +881,7 @@ fn main() { | |||
| let starting_slot = value_t_or_exit!(arg_matches, "starting_slot", Slot); | |||
| let allow_dead_slots = arg_matches.is_present("allow_dead_slots"); | |||
| output_ledger( | |||
| open_blockstore(&ledger_path), | |||
| open_blockstore(&ledger_path, AccessType::AllowSecondary), | |||
There was a problem hiding this comment.
TODO: carefully audit for the appropriate AccessType for all of subcommands.
There was a problem hiding this comment.
Done, I've tested all the subcommands and they worked as intended.
@mvines FYI, I found graph to be very useful to create a graph right from a still running validator's ledger. :)
There was a problem hiding this comment.
totally, this is gonna be yuuuuge!
ryoqun
commented
May 24, 2020
|
|
CI is broken... ;) |
ryoqun
commented
May 24, 2020
ryoqun
commented
May 24, 2020
|
TODO: maybe don't fallback by default; instead explicit |
This was referenced May 24, 2020
|
This looks very promising, thanks @ryoqun |
ryoqun
force-pushed
the
rocksdb-secondary-open
branch
from
3be691a to
e88a3c9
Compare
Codecov Report
@@ Coverage Diff @@
## master #10209 +/- ##
=========================================
- Coverage 81.3% 81.3% -0.1%
=========================================
Files 288 288
Lines 67000 67135 +135
=========================================
+ Hits 54510 54603 +93
- Misses 12490 12532 +42 |
mvines
reviewed
May 25, 2020
I've tested enough. Be confident to start using the secondary fallback by default. |
ryoqun
changed the title
ryoqun
force-pushed
the
rocksdb-secondary-open
branch
from
1042882 to
890e823
Compare
|
@mvines This is ready for review!! :) |
mvines
reviewed
Jun 1, 2020
ledger/src/blockstore.rs
Outdated
| Self::do_open_with_signal(ledger_path, AccessType::OnlyPrimary) | ||
| } | ||
|
|
||
| pub fn open_with_signal_with_access_type( |
There was a problem hiding this comment.
Suggested change
| pub fn open_with_signal_with_access_type( | |
| pub fn open_with_signal_and_access_type( |
@mvines Yeah, I deliberately do so because...:
|
Co-authored-by: Michael Vines <[email protected]>
mvines
reviewed
Jun 1, 2020
mvines
previously approved these changes
Jun 2, 2020
mergify bot
pushed a commit
that referenced
this pull request
Jun 3, 2020
automerge (cherry picked from commit caa7f7a) # Conflicts: # ledger-tool/Cargo.toml # ledger-tool/src/main.rs
mergify bot
pushed a commit
that referenced
this pull request
Jun 3, 2020
automerge (cherry picked from commit caa7f7a)
solana-grimes
pushed a commit
that referenced
this pull request
Jun 3, 2020
solana-grimes
pushed a commit
that referenced
this pull request
Jun 3, 2020
This was referenced Jun 3, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
it seems that the rocksdb's notorious limitation is now lifted: facebook/rocksdb#4899 ㊗️
Maybe our bitter experience caused blindness for the rocksdb; it's certainly improving. ;)
Summary of Changes
Fallback to secondary access when primary failed from
ledget-tool's read-only subcommands.solana-validatorand other subcommands behavior hasn't been changed.This realizes to run
create-snapshotwhile a validator is running and other various debugging subcommands.PS: I know my half-backed
ledger-toolpr #9490 is laying around... Sorry for our longer pr queue. ;) I thought this worth to interrupt. :)