Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

merge queue: embarking main (767c35f) and #3578 together #3661

Closed
wants to merge 59 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
ea41888
Implement phase 3 of shielded sync
batconjurer Jul 9, 2024
be08bce
Generalize the shutdown signal abstraction in the sdk
sug0 Jul 29, 2024
684887b
Add progress trackers to ss
sug0 Jul 29, 2024
7e292f0
Start ss from last query height
sug0 Jul 30, 2024
a861234
Increment ss progress bars by given block ranges
sug0 Jul 30, 2024
bb5682a
Remove ss start height param
sug0 Jul 30, 2024
d27e5a9
Always start ss from min vk height
sug0 Jul 30, 2024
5828a4d
Set correct upper bound on ss progress bars
sug0 Jul 30, 2024
746aed7
Limit number of concurrent ledger client requests
sug0 Jul 30, 2024
6b3d8d4
Limit number of concurrent indexer client requests
sug0 Jul 30, 2024
0bb28e8
Remove async from extract_masp_tx
sug0 Jul 30, 2024
3f7e9ef
Improve pre_cache_transaction
sug0 Jul 30, 2024
51a22ca
Remove scan_tx
sug0 Jul 30, 2024
bac111a
Allow ss fetch to run in wasm
sug0 Jul 30, 2024
2b60270
Remove ss specific TODOs
sug0 Jul 30, 2024
e2cd22a
Spelling
sug0 Jul 30, 2024
26095ef
Fix note positions bug in ss
sug0 Jul 31, 2024
4639363
Only store valid notes and their offsets
sug0 Jul 31, 2024
f239ca1
Make shutdown signal aware of whether a tokio rt is running
sug0 Aug 1, 2024
79d612c
Fixing unit tests
batconjurer Aug 1, 2024
75f7ea0
Cleanups and test fixes
batconjurer Aug 1, 2024
f150969
Improved progress bars
batconjurer Aug 1, 2024
d3a7608
Cleanup dependencies
sug0 Aug 1, 2024
981d784
Fetch latest block index from masp indexer
sug0 Aug 1, 2024
cefc882
Skip fetching blocks without masp txs in indexer client
sug0 Aug 1, 2024
f661630
Make block index calls optional
sug0 Aug 1, 2024
5ab214a
Bump `init_once`
sug0 Aug 1, 2024
42fca85
Cleanups after rebasing on main
batconjurer Aug 2, 2024
d2b2068
fixed checked crates
batconjurer Aug 2, 2024
d3807cc
Bump `init_once`
sug0 Aug 2, 2024
7e30b94
Implement shielded sync cache IO for benches
sug0 Aug 2, 2024
92e64cc
Bump `init_once`
sug0 Aug 5, 2024
ca3e993
Wait until ss target height is available
sug0 Aug 7, 2024
c0fcf87
Wait for the channel to drain
sug0 Aug 8, 2024
3b8f55f
Unlock rayon dependency
sug0 Aug 8, 2024
c2c9c43
Clone rpc client
sug0 Aug 8, 2024
87546ff
Fetch blocks not accounted for by the masp txs block index
sug0 Aug 9, 2024
e77a00d
Print progress on applied scanned txs
sug0 Aug 9, 2024
6c6bf7a
Optionally announce exit on interrupt signal
sug0 Aug 9, 2024
622bc8d
Replace tracing calls with bar specific io
sug0 Aug 9, 2024
e2cb092
Add shutdown signal to shielded sync config
sug0 Aug 9, 2024
528bd32
Expose max concurrent fetches
sug0 Aug 9, 2024
c1734e1
Tested the block filter and fixed an off-by-one error
batconjurer Aug 12, 2024
a61ba30
Added retry strategy to shielded sync args
batconjurer Aug 12, 2024
01934d3
Tiny
batconjurer Aug 13, 2024
8561699
Renamed tx_note_map to note_index
batconjurer Aug 13, 2024
715a7d4
Renamed main ss method from fetch to sync
batconjurer Aug 13, 2024
b06673a
Remaining notes map rename
sug0 Aug 13, 2024
90b7478
Simplify IntoIter impl
sug0 Aug 13, 2024
9a887e1
Fetch concurrently within masp tx indexer client rpc method
sug0 Aug 13, 2024
8293cc4
Fix CLI formatting
sug0 Aug 13, 2024
e0d5287
Undo accidental find-replaces
sug0 Aug 13, 2024
2804a82
Avoid redundant `is_err` check
sug0 Aug 13, 2024
deb3e43
Fix error string
sug0 Aug 13, 2024
d09f4b8
Fix benchmarks
sug0 Aug 14, 2024
fa6ecf8
Fix inaccessible indexer client
sug0 Aug 14, 2024
b689533
Changelog for #3578
sug0 Aug 2, 2024
9626233
Merge remote-tracking branch 'origin/main' into tiago+jacob/masp-ss-c…
sug0 Aug 16, 2024
2cedd4c
Merge of #3578
mergify[bot] Aug 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .changelog/unreleased/SDK/3578-masp-ss-client-phase-3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
- Parallelize the shielded sync implementation in Namada.
([\#3578](https://github.com/anoma/namada/pull/3578))
57 changes: 55 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 6 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ expectrl = "0.7.0"
eyre = "0.6.12"
fd-lock = "3.0.12"
flate2 = "1.0.22"
flume = "0.11.0"
fs_extra = "1.2.0"
futures = "0.3"
git2 = { version = "0.18.1", default-features = false }
Expand All @@ -118,9 +119,11 @@ ibc-testkit = { git = "https://github.com/heliaxdev/cosmos-ibc-rs", rev = "1dd9b
ics23 = "0.11.0"
index-set = { git = "https://github.com/heliaxdev/index-set", tag = "v0.8.1", features = ["serialize-borsh", "serialize-serde"] }
indexmap = { git = "https://github.com/heliaxdev/indexmap", tag = "2.2.4-heliax-1", features = ["borsh-schema", "serde"] }
init-once = "0.6.0"
itertools = "0.12.1"
jubjub = "0.10"
k256 = { version = "0.13.0", default-features = false, features = ["ecdsa", "pkcs8", "precomputed-tables", "serde", "std"]}
kdam = "0.5.2"
konst = { version = "0.3.8", default-features = false }
lazy_static = "1.4.0"
# TODO: upstreamed in https://github.com/ledger-community/rust-ledger/pull/9
Expand Down Expand Up @@ -152,7 +155,7 @@ prost = "0.12.0"
prost-types = "0.12.0"
rand = {version = "0.8", default-features = false}
rand_core = {version = "0.6", default-features = false}
rayon = "=1.5.3"
rayon = "1.5.3"
regex = "1.10.2"
reqwest = "0.11.4"
ripemd = "0.1"
Expand Down Expand Up @@ -194,6 +197,7 @@ tracing = "0.1.30"
tracing-appender = "0.2.2"
tracing-log = "0.2.0"
tracing-subscriber = {version = "0.3.7", default-features = false, features = ["env-filter", "fmt"]}
typed-builder = "0.19.1"
wasmparser = "0.107.0"
wasm-instrument = {version = "0.4.0", features = ["sign_ext"]}
wasmer = "4.3.5"
Expand All @@ -204,6 +208,7 @@ wasmer-types = "4.3.5"
wasmer-vm = "4.3.5"
wasmtimer = "0.2.0"
winapi = "0.3.9"
xorf = { version = "0.11.0", features = ["serde"] }
yansi = "0.5.1"
zeroize = { version = "1.5.5", features = ["zeroize_derive"] }

Expand Down
1 change: 1 addition & 0 deletions crates/apps_lib/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ fd-lock.workspace = true
flate2.workspace = true
futures.workspace = true
itertools.workspace = true
kdam.workspace = true
lazy_static = { workspace = true, optional = true }
linkme = { workspace = true, optional = true }
ledger-lib = { workspace = true }
Expand Down
58 changes: 32 additions & 26 deletions crates/apps_lib/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3165,6 +3165,7 @@ pub mod args {
use namada_sdk::ibc::core::host::types::identifiers::{ChannelId, PortId};
use namada_sdk::keccak::KeccakHash;
use namada_sdk::key::*;
use namada_sdk::masp::utils::RetryStrategy;
use namada_sdk::masp::{MaspEpoch, PaymentAddress};
use namada_sdk::storage::{self, BlockHeight, Epoch};
use namada_sdk::time::DateTimeUtc;
Expand Down Expand Up @@ -3214,8 +3215,6 @@ pub mod args {
pub const BIRTHDAY: ArgOpt<BlockHeight> = arg_opt("birthday");
pub const BLOCK_HEIGHT: Arg<BlockHeight> = arg("block-height");
pub const BLOCK_HEIGHT_OPT: ArgOpt<BlockHeight> = arg_opt("height");
pub const BLOCK_HEIGHT_FROM_OPT: ArgOpt<BlockHeight> =
arg_opt("from-height");
pub const BLOCK_HEIGHT_TO_OPT: ArgOpt<BlockHeight> = arg_opt("to-height");
pub const BRIDGE_POOL_GAS_AMOUNT: ArgDefault<token::DenominatedAmount> =
arg_default(
Expand Down Expand Up @@ -3353,6 +3352,8 @@ pub mod args {
pub const MASP_EPOCH: ArgOpt<MaspEpoch> = arg_opt("masp-epoch");
pub const MAX_COMMISSION_RATE_CHANGE: Arg<Dec> =
arg("max-commission-rate-change");
pub const MAX_CONCURRENT_FETCHES: ArgDefault<usize> =
arg_default("max-concurrent-fetches", DefaultFn(|| 100));
pub const MAX_ETH_GAS: ArgOpt<u64> = arg_opt("max_eth-gas");
pub const MEMO_OPT: ArgOpt<String> = arg_opt("memo");
pub const MIGRATION_PATH: ArgOpt<PathBuf> = arg_opt("migration-path");
Expand Down Expand Up @@ -3406,7 +3407,7 @@ pub mod args {
pub const REFUND_TARGET: ArgOpt<WalletTransferTarget> =
arg_opt("refund-target");
pub const RELAYER: Arg<Address> = arg("relayer");
pub const SAFE_MODE: ArgFlag = flag("safe-mode");
pub const RETRIES: ArgOpt<u64> = arg_opt("retries");
pub const SCHEME: ArgDefault<SchemeType> =
arg_default("scheme", DefaultFn(|| SchemeType::Ed25519));
pub const SHELL: Arg<Shell> = arg("shell");
Expand Down Expand Up @@ -3466,6 +3467,8 @@ pub mod args {
pub const VIEWING_KEYS: ArgMulti<WalletViewingKey, GlobStar> =
arg_multi("viewing-keys");
pub const VP: ArgOpt<String> = arg_opt("vp");
pub const WAIT_FOR_LAST_QUERY_HEIGHT: ArgFlag =
flag("wait-for-last-query-height");
pub const WALLET_ALIAS_FORCE: ArgFlag = flag("wallet-alias-force");
pub const WASM_CHECKSUMS_PATH: Arg<PathBuf> = arg("wasm-checksums-path");
pub const WASM_DIR: ArgOpt<PathBuf> = arg_opt("wasm-dir");
Expand Down Expand Up @@ -4099,14 +4102,12 @@ pub mod args {
gas_price: self.gas_price,
eth_addr: self.eth_addr,
sync: self.sync,
safe_mode: self.safe_mode,
}
}
}

impl Args for RelayBridgePoolProof<CliTypes> {
fn parse(matches: &ArgMatches) -> Self {
let safe_mode = SAFE_MODE.parse(matches);
let ledger_address = LEDGER_ADDRESS.parse(matches);
let hashes = HASH_LIST.parse(matches);
let relayer = RELAYER.parse(matches);
Expand Down Expand Up @@ -4137,16 +4138,11 @@ pub mod args {
eth_rpc_endpoint,
eth_addr,
confirmations,
safe_mode,
}
}

fn def(app: App) -> App {
app.arg(LEDGER_ADDRESS.def().help(LEDGER_ADDRESS_ABOUT))
.arg(SAFE_MODE.def().help(wrap!(
"Safe mode overrides keyboard interrupt signals, to \
ensure Ethereum transfers aren't canceled midway through."
)))
.arg(HASH_LIST.def().help(wrap!(
"Whitespace separated Keccak hash list of transfers in \
the Bridge pool."
Expand Down Expand Up @@ -4282,15 +4278,13 @@ pub mod args {
sync: self.sync,
retry_dur: self.retry_dur,
success_dur: self.success_dur,
safe_mode: self.safe_mode,
}
}
}

impl Args for ValidatorSetUpdateRelay<CliTypes> {
fn parse(matches: &ArgMatches) -> Self {
let ledger_address = LEDGER_ADDRESS.parse(matches);
let safe_mode = SAFE_MODE.parse(matches);
let daemon = DAEMON_MODE.parse(matches);
let epoch = EPOCH.parse(matches);
let gas = ETH_GAS.parse(matches);
Expand All @@ -4315,16 +4309,11 @@ pub mod args {
eth_addr,
retry_dur,
success_dur,
safe_mode,
}
}

fn def(app: App) -> App {
app.arg(LEDGER_ADDRESS.def().help(LEDGER_ADDRESS_ABOUT))
.arg(SAFE_MODE.def().help(wrap!(
"Safe mode overrides keyboard interrupt signals, to \
ensure Ethereum transfers aren't canceled midway through."
)))
.arg(DAEMON_MODE.def().help(wrap!(
"Run in daemon mode, which will continuously perform \
validator set updates."
Expand Down Expand Up @@ -6601,18 +6590,26 @@ pub mod args {
impl Args for ShieldedSync<CliTypes> {
fn parse(matches: &ArgMatches) -> Self {
let ledger_address = CONFIG_RPC_LEDGER_ADDRESS.parse(matches);
let start_query_height = BLOCK_HEIGHT_FROM_OPT.parse(matches);
let last_query_height = BLOCK_HEIGHT_TO_OPT.parse(matches);
let spending_keys = DATED_SPENDING_KEYS.parse(matches);
let viewing_keys = DATED_VIEWING_KEYS.parse(matches);
let with_indexer = WITH_INDEXER.parse(matches);
let wait_for_last_query_height =
WAIT_FOR_LAST_QUERY_HEIGHT.parse(matches);
let max_concurrent_fetches = MAX_CONCURRENT_FETCHES.parse(matches);
let retry_strategy = match RETRIES.parse(matches) {
Some(times) => RetryStrategy::Times(times),
None => RetryStrategy::Forever,
};
Self {
ledger_address,
start_query_height,
last_query_height,
spending_keys,
viewing_keys,
with_indexer,
wait_for_last_query_height,
max_concurrent_fetches,
retry_strategy,
}
}

Expand All @@ -6621,11 +6618,6 @@ pub mod args {
.arg(BLOCK_HEIGHT_TO_OPT.def().help(wrap!(
"Option block height to sync up to. Default is latest."
)))
.arg(
BLOCK_HEIGHT_FROM_OPT
.def()
.help(wrap!("Option block height to sync from.")),
)
.arg(DATED_SPENDING_KEYS.def().help(wrap!(
"List of new spending keys with which to check note \
ownership. These will be added to the shielded context. \
Expand All @@ -6643,6 +6635,18 @@ pub mod args {
present, the shielded sync will be performed using data \
retrieved from the given indexer."
)))
.arg(WAIT_FOR_LAST_QUERY_HEIGHT.def().help(wrap!(
"Wait until the last height to sync is available instead \
of returning early from the shielded sync."
)))
.arg(MAX_CONCURRENT_FETCHES.def().help(wrap!(
"Maximum number of fetch jobs that will ever execute \
concurrently during the shielded sync."
)))
.arg(RETRIES.def().help(wrap!(
"Maximum number of times to retry fetching. If no \
argument is provided, defaults to retrying forever."
)))
}
}

Expand All @@ -6656,8 +6660,9 @@ pub mod args {
let chain_ctx = ctx.borrow_mut_chain_or_exit();

Ok(ShieldedSync {
max_concurrent_fetches: self.max_concurrent_fetches,
wait_for_last_query_height: self.wait_for_last_query_height,
ledger_address: chain_ctx.get(&self.ledger_address),
start_query_height: self.start_query_height,
last_query_height: self.last_query_height,
spending_keys: self
.spending_keys
Expand All @@ -6669,7 +6674,8 @@ pub mod args {
.iter()
.map(|vk| chain_ctx.get_cached(vk))
.collect(),
with_indexer: self.with_indexer.map(|_| ()),
with_indexer: self.with_indexer,
retry_strategy: self.retry_strategy,
})
}
}
Expand Down
2 changes: 1 addition & 1 deletion crates/apps_lib/src/cli/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ use crate::facade::tendermint_rpc::{HttpClient, Url as TendermintUrl};

/// Trait for clients that can be used with the CLI.
#[async_trait::async_trait(?Send)]
pub trait CliClient: Client + Sync {
pub trait CliClient: Client + Send + Sync + 'static {
fn from_tendermint_address(address: &TendermintUrl) -> Self;
async fn wait_until_node_is_synced(
&self,
Expand Down
Loading
Loading