Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(db-tool): Tool to run DB migrations #9333

Merged
merged 59 commits into from
Jul 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
5b14f00
fix(db-tool): Tool to run DB migrations
nikurt Jul 20, 2023
e51a689
feat: simple nightshade v2 - shard layout with 5 shards (#9274)
wacban Jul 13, 2023
832f0c3
refactor: small refactorings and improvements (#9296)
wacban Jul 13, 2023
d36b1c3
refactor: refactoring and commenting some resharding code (#9299)
wacban Jul 14, 2023
de0d9ba
near-vm-runner: move protocol-sensitive error schemas to near-primiti…
nagisa Jul 14, 2023
aee8ab3
rust: 1.70.0 -> 1.71.0 (#9302)
nagisa Jul 14, 2023
2981f5d
fix(state-sync): Always use flat storage when catching up (#9311)
nikurt Jul 14, 2023
2ab8fcc
fix(state-snapshot): Tool to make DB snapshots (#9308)
nikurt Jul 14, 2023
a3c8ab5
chore(estimator): remove TTN read estimation (#9307)
jakmeier Jul 14, 2023
a6684d0
feat: expose more RocksDB properties (#9279)
pugachAG Jul 14, 2023
8ba86ab
chain: remove deprecated near_peer_message_received_total metric (#9312)
mina86 Jul 14, 2023
0318ff8
refactor: improvements to logging (#9309)
wacban Jul 15, 2023
04b630a
nearcore: remove old deprecation notice about network.external_addres…
mina86 Jul 17, 2023
d707f24
fix(state-sync): Test showing that state sync can't always generate s…
nikurt Jul 17, 2023
8e7262a
fix(locust): wait for base on_locust_init() to finish before other in…
marcelo-gonzalez Jul 17, 2023
8402f75
fix(state-sync): Simplify storage format of state sync dump progress …
nikurt Jul 17, 2023
86ea6e3
Fix proxy-based nayduck tests so that they can run on non-unix system…
robin-near Jul 17, 2023
2df7a68
fix: fixed nayduck test state_sync_fail.py for nightly build (#9320)
wacban Jul 17, 2023
33b15e1
feat: add database tool subcommand for State read perf testing (#9276)
pugachAG Jul 17, 2023
edb3d79
RoutingTable V2: Distance Vector Routing (#9187)
saketh-are Jul 18, 2023
612348a
fix: use logging instead of print statements (#9277)
itsyaasir Jul 18, 2023
c935186
refactor: todo to remove flat storage creation parameters (#9250)
Longarithm Jul 18, 2023
e254ae8
refactor(loadtest): backwards compatible type hints (#9323)
jakmeier Jul 18, 2023
365c69a
feat(state-sync): Add config for number of downloads during catchup (…
VanBarbascu Jul 18, 2023
1477ec8
chore: Update RocksDB to 0.21 (#9298)
Jul 20, 2023
4c76e19
fix(db-tool): Tool to run DB migrations
nikurt Jul 20, 2023
c5a85e8
fix(db-tool): Tool to run DB migrations
nikurt Jul 20, 2023
909b1f0
Merge branch 'master' into nikurt-tool-db-migration
nikurt Jul 20, 2023
dcb483d
fix(db-tool): Tool to run DB migrations
nikurt Jul 20, 2023
a1f5fe8
fmt
nikurt Jul 20, 2023
e1db9eb
Merge master into nikurt-tool-db-migration
near-bulldozer[bot] Jul 20, 2023
5b3f643
Merge refs/heads/master into nikurt-tool-db-migration
near-bulldozer[bot] Jul 20, 2023
22e7a57
Merge refs/heads/master into nikurt-tool-db-migration
near-bulldozer[bot] Jul 20, 2023
00cb7b1
Merge refs/heads/master into nikurt-tool-db-migration
near-bulldozer[bot] Jul 20, 2023
b5a852c
Merge refs/heads/master into nikurt-tool-db-migration
near-bulldozer[bot] Jul 21, 2023
929b7af
Merge refs/heads/master into nikurt-tool-db-migration
near-bulldozer[bot] Jul 21, 2023
a5f1e31
Merge refs/heads/master into nikurt-tool-db-migration
near-bulldozer[bot] Jul 21, 2023
24ebda2
fmt
nikurt Jul 24, 2023
7689ebc
Merge refs/heads/master into nikurt-tool-db-migration
near-bulldozer[bot] Jul 24, 2023
12a8e6f
fix(db-tool): Tool to run DB migrations
nikurt Jul 20, 2023
18c2333
feat: simple nightshade v2 - shard layout with 5 shards (#9274)
wacban Jul 13, 2023
3dbc17d
refactor: small refactorings and improvements (#9296)
wacban Jul 13, 2023
44fbef5
refactor: refactoring and commenting some resharding code (#9299)
wacban Jul 14, 2023
408e211
rust: 1.70.0 -> 1.71.0 (#9302)
nagisa Jul 14, 2023
7419633
fix(state-sync): Always use flat storage when catching up (#9311)
nikurt Jul 14, 2023
c972d75
fix(state-snapshot): Tool to make DB snapshots (#9308)
nikurt Jul 14, 2023
03e7c0a
refactor: improvements to logging (#9309)
wacban Jul 15, 2023
798a4a3
fix(state-sync): Test showing that state sync can't always generate s…
nikurt Jul 17, 2023
a98274d
feat: add database tool subcommand for State read perf testing (#9276)
pugachAG Jul 17, 2023
76127bf
RoutingTable V2: Distance Vector Routing (#9187)
saketh-are Jul 18, 2023
5eb5c8a
feat(state-sync): Add config for number of downloads during catchup (…
VanBarbascu Jul 18, 2023
232d8a6
Merge
nikurt Jul 24, 2023
578e1d7
Merge
nikurt Jul 24, 2023
46bd0c0
fmt
nikurt Jul 24, 2023
2687ca3
fmt
nikurt Jul 24, 2023
bef509c
fmt
nikurt Jul 24, 2023
dae314b
fmt
nikurt Jul 24, 2023
6834636
fmt
nikurt Jul 24, 2023
94ac6df
fmt
nikurt Jul 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions tools/database/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,18 @@ available in `/home/ubuntu/.near/data/snapshot`
This command can be helpful before attempting activities that can potentially
corrupt the database.

### Run DB Migrations

Opens the DB and runs migrations to bring it to the actual version expected by `neard`
Example usage:
```bash
cargo run --bin neard database run-migrations
```

For example, if the binary expects DB version `38`, but the DB is currently
version `36`, the command will open the DB, run migrations that bring the DB
from version `36` to version `38`, and then exits.

## State read perf
A tool for performance testing hot storage RocksDB State column reads.
Use help to get more details: `neard database state-perf --help`
5 changes: 5 additions & 0 deletions tools/database/src/commands.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
use crate::adjust_database::ChangeDbKindCommand;
use crate::analyse_data_size_distribution::AnalyseDataSizeDistributionCommand;
use crate::make_snapshot::MakeSnapshotCommand;
use crate::run_migrations::RunMigrationsCommand;
use crate::state_perf::StatePerfCommand;
use clap::Parser;
use std::path::PathBuf;
Expand All @@ -23,6 +24,9 @@ enum SubCommand {
/// Make snapshot of the database
MakeSnapshot(MakeSnapshotCommand),

/// Run migrations,
RunMigrations(RunMigrationsCommand),

/// Run performance test for State column reads.
/// Uses RocksDB data specified via --home argument.
StatePerf(StatePerfCommand),
Expand All @@ -41,6 +45,7 @@ impl DatabaseCommand {
.unwrap_or_else(|e| panic!("Error loading config: {:#}", e));
cmd.run(home, near_config.config.archive, &near_config.config.store)
}
SubCommand::RunMigrations(cmd) => cmd.run(home),
SubCommand::StatePerf(cmd) => cmd.run(home),
}
}
Expand Down
1 change: 1 addition & 0 deletions tools/database/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@ mod adjust_database;
mod analyse_data_size_distribution;
pub mod commands;
mod make_snapshot;
mod run_migrations;
mod state_perf;
mod utils;
16 changes: 16 additions & 0 deletions tools/database/src/run_migrations.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
use std::path::Path;

#[derive(clap::Args)]
pub(crate) struct RunMigrationsCommand {}

impl RunMigrationsCommand {
pub(crate) fn run(&self, home_dir: &Path) -> anyhow::Result<()> {
let mut near_config = nearcore::config::load_config(
&home_dir,
near_chain_configs::GenesisValidationMode::UnsafeFast,
)
.unwrap_or_else(|e| panic!("Error loading config: {:#}", e));
nearcore::open_storage(home_dir, &mut near_config)?;
Ok(())
}
}