Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(node): Port (most of) Node to the Node Framework #2196

Merged
merged 51 commits into from
Jun 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
62414e6
Snapshot
popzxc Jun 5, 2024
63faae8
Merge branch 'main' into popzxc-en-framework
popzxc Jun 5, 2024
b31ae56
Basic skeleton
popzxc Jun 5, 2024
43b6ca3
Merge branch 'main' into popzxc-en-framework
popzxc Jun 6, 2024
0f4b9fb
Turns out, my code didn't even compile. Huh
popzxc Jun 6, 2024
3a7f84b
Integrate builder into EN
popzxc Jun 6, 2024
acacea4
Extract postgres metrics into a separate layer
popzxc Jun 6, 2024
ec0ec74
Add main node client layer
popzxc Jun 6, 2024
d90b0d2
Instantiate main node client healthcheck in the corresponding layer
popzxc Jun 6, 2024
9a52358
Spawn health check server
popzxc Jun 6, 2024
6d93bfd
Add prometheus exporter layer
popzxc Jun 6, 2024
81f66a2
Merge branch 'main' into popzxc-en-framework
popzxc Jun 7, 2024
ab2ffa1
Add tree data fetcher layer
popzxc Jun 7, 2024
b58c4ef
Add L1 client layer
popzxc Jun 7, 2024
e167d72
Improve reorg detector precondition & ensure that the tree runs with …
popzxc Jun 10, 2024
e3883dd
Extract persistence from MempoolIO layer
popzxc Jun 10, 2024
9728332
Tweak persistence layer to support ENs
popzxc Jun 10, 2024
c3e59fb
Implement external IO layer
popzxc Jun 10, 2024
5afe04b
Remove config dependency from batch executor layer
popzxc Jun 10, 2024
70ce720
Remove config dependency from state keeper layer
popzxc Jun 10, 2024
d0218ba
Improve docs for batch executor
popzxc Jun 10, 2024
77402b5
Allow to override batch executor bytecode compression in the layer
popzxc Jun 10, 2024
9d3fdab
Integrate state keeper and consensus
popzxc Jun 10, 2024
ecd1648
Integrate pruning
popzxc Jun 10, 2024
0c1bfc4
Integrate commitment mode validation
popzxc Jun 10, 2024
a567a38
Integrate consistency checker layer
popzxc Jun 10, 2024
ebcb0c0
Integrate consistency checker layer
popzxc Jun 10, 2024
84b75de
Formatting
popzxc Jun 10, 2024
4fa77cf
Add commitment generator layer
popzxc Jun 10, 2024
9dd29a0
Merge branch 'main' into popzxc-en-framework
popzxc Jun 11, 2024
405dcf2
Merge branch 'main' into popzxc-en-framework
popzxc Jun 12, 2024
619d6fc
Add validate chain ids task
popzxc Jun 12, 2024
9cc724a
Add batch status updater layer
popzxc Jun 12, 2024
2c172ee
Run sync state updater
popzxc Jun 12, 2024
02a16f7
Integrate metadata calculator
popzxc Jun 12, 2024
a26922c
Support whitelisted_tokens_for_aa_cache updater
popzxc Jun 12, 2024
00fffb1
Add several API layers
popzxc Jun 12, 2024
041f439
Spawn tx sender in EN
popzxc Jun 13, 2024
99176be
Add account nonce sweeper task
popzxc Jun 13, 2024
75fec58
Merge branch 'main' into popzxc-en-framework
popzxc Jun 13, 2024
8b433bf
Merge fixups
popzxc Jun 13, 2024
2ba8153
Main node fee params fetcher layer
popzxc Jun 13, 2024
c939390
Support pruning_info_refresh_interval in API
popzxc Jun 13, 2024
2d9180f
Integrate API layers
popzxc Jun 13, 2024
c68be9f
Integrate extended tracing
popzxc Jun 13, 2024
b30b71b
Make framework-based EN run
popzxc Jun 13, 2024
c75bf31
whitelist whitelisted
popzxc Jun 13, 2024
fac9cba
Merge branch 'main' into popzxc-en-framework
popzxc Jun 19, 2024
6fded6f
Make postgres scraping task unconstrained
popzxc Jun 19, 2024
397d7a1
Make prometheus exporter unconstrained
popzxc Jun 19, 2024
0f1ee20
Misc comment fixes
popzxc Jun 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions checks-config/era.dic
Original file line number Diff line number Diff line change
Expand Up @@ -962,6 +962,7 @@ zksync_merkle_tree
TreeMetadata
delegator
decrement
whitelisted
Bbellman
Sbellman
DCMAKE
Expand Down
1 change: 1 addition & 0 deletions core/bin/external_node/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ zksync_metadata_calculator.workspace = true
zksync_node_sync.workspace = true
zksync_node_api_server.workspace = true
zksync_node_consensus.workspace = true
zksync_node_framework.workspace = true
vlog.workspace = true

zksync_concurrency.workspace = true
Expand Down
31 changes: 27 additions & 4 deletions core/bin/external_node/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ use std::{collections::HashSet, net::Ipv4Addr, str::FromStr, sync::Arc, time::Du
use anyhow::Context as _;
use clap::Parser;
use metrics::EN_METRICS;
use node_builder::ExternalNodeBuilder;
use tokio::{
sync::{oneshot, watch, RwLock},
task::{self, JoinHandle},
Expand Down Expand Up @@ -63,6 +64,7 @@ mod config;
mod init;
mod metadata;
mod metrics;
mod node_builder;
#[cfg(test)]
mod tests;

Expand Down Expand Up @@ -426,10 +428,11 @@ async fn run_api(
.build()
.await
.context("failed to build a proxy_cache_updater_pool")?;
task_handles.push(tokio::spawn(tx_proxy.run_account_nonce_sweeper(
proxy_cache_updater_pool.clone(),
stop_receiver.clone(),
)));
task_handles.push(tokio::spawn(
tx_proxy
.account_nonce_sweeper_task(proxy_cache_updater_pool.clone())
.run(stop_receiver.clone()),
));

let fee_params_fetcher_handle =
tokio::spawn(fee_params_fetcher.clone().run(stop_receiver.clone()));
Expand Down Expand Up @@ -701,6 +704,10 @@ struct Cli {
/// Comma-separated list of components to launch.
#[arg(long, default_value = "all")]
components: ComponentsToRun,

/// Run the node using the node framework.
#[arg(long)]
use_node_framework: bool,
}

#[derive(Debug, Clone, Copy, PartialEq, Hash, Eq)]
Expand Down Expand Up @@ -784,6 +791,22 @@ async fn main() -> anyhow::Result<()> {
.fetch_remote(main_node_client.as_ref())
.await
.context("failed fetching remote part of node config from main node")?;

// If the node framework is used, run the node.
if opt.use_node_framework {
// We run the node from a different thread, since the current thread is in tokio context.
std::thread::spawn(move || {
let node =
ExternalNodeBuilder::new(config).build(opt.components.0.into_iter().collect())?;
node.run()?;
anyhow::Ok(())
})
.join()
.expect("Failed to run the node")?;

return Ok(());
}

if let Some(threshold) = config.optional.slow_query_threshold() {
ConnectionPool::<Core>::global_config().set_slow_query_threshold(threshold)?;
}
Expand Down
Loading
Loading