Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(Prover CLI): status batch command #1638

Merged
merged 84 commits into from
May 3, 2024
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
17e564c
fmt
ColoCarletti Apr 8, 2024
245b0a4
wip
ColoCarletti Apr 8, 2024
8e84243
add cargo.toml cargo.lock
ColoCarletti Apr 8, 2024
17517b1
go back to anihow
ColoCarletti Apr 8, 2024
4ab7ce8
fmt
ColoCarletti Apr 8, 2024
4655201
rm tool workspace
ColoCarletti Apr 9, 2024
c32d1aa
fmt
ColoCarletti Apr 10, 2024
8e49e04
zk fmt
ColoCarletti Apr 10, 2024
80e13da
fix typo
ColoCarletti Apr 10, 2024
0f2ebd8
move query to prover_dal
ColoCarletti Apr 12, 2024
69d2104
remove --all flag
Apr 15, 2024
c9e6e5e
change command name
ColoCarletti Apr 16, 2024
83fab40
merge main
ColoCarletti Apr 16, 2024
cdbb467
fmt
ColoCarletti Apr 16, 2024
d1f3994
add sqlx query json
ColoCarletti Apr 16, 2024
4ee3ff2
Merge branch 'main' into prover_cli_status_command
ilitteri Apr 16, 2024
fff3910
update README
ColoCarletti Apr 16, 2024
ef906b9
zk fmt
ColoCarletti Apr 16, 2024
3758547
Merge branch 'prover_cli_status_command' of github.com:matter-labs/zk…
ColoCarletti Apr 16, 2024
b3051b2
remove eol
ColoCarletti Apr 16, 2024
ee9d35b
Move jobs cmd to status cmd module
Apr 16, 2024
82faf32
Refactor status cmd
Apr 16, 2024
aa44864
Update Cargo.lock
Apr 16, 2024
3fe32b6
zk fmt
Apr 16, 2024
d43dd15
Merge pull request #1705 from matter-labs/il-refactor-prover-cli-stat…
ilitteri Apr 17, 2024
70b0348
Merge branch 'main' into prover_cli_status_command
ilitteri Apr 17, 2024
585059b
Merge branch 'main' into prover_cli_status_command
ilitteri Apr 17, 2024
9977e95
Refactor
Apr 18, 2024
ee88cfe
Merge branch 'main' into prover_cli_status_command
ilitteri Apr 18, 2024
075e6dc
Merge branch 'main' into prover_cli_status_command
ColoCarletti Apr 19, 2024
0120a09
fix typo
ColoCarletti Apr 19, 2024
9a11a94
modify query to include aggregation_round
ColoCarletti Apr 19, 2024
1c92e95
change query return type
ColoCarletti Apr 24, 2024
05e1636
Update Cargo files
Apr 24, 2024
58d47ad
Setle status cmd baseline
Apr 24, 2024
f778759
Add status cmd utils
Apr 24, 2024
e945df2
Add status utils
Apr 24, 2024
6585378
zk fmt
Apr 24, 2024
47df898
Document enums
Apr 24, 2024
ceae289
Add BatchDataBuilder struct
Apr 24, 2024
f6d7b9a
Add TaskStatus::Custom enum variant
Apr 24, 2024
8e16d35
Fix BatchDataBuilder
Apr 24, 2024
a76cb32
Rollback
Apr 24, 2024
c460029
impl From<ProofCompressionJobStatus> for TaskStatus
Apr 24, 2024
e76da1f
Fix TaskStatus::Custom fmt
Apr 24, 2024
4be214e
Add query for getting proof compression job info for a batch
Apr 25, 2024
e9e197f
Add query for getting proof compression job info for a batch
Apr 25, 2024
7c9a499
Handle proof compression job status for batches
Apr 25, 2024
17b5875
Fix get_proof_compression_job_status_for_batch
Apr 25, 2024
3fdc055
Remove BatchDataBuilder struct
Apr 25, 2024
a2ad3f9
Merge branch 'prover_cli_status_command' of github.com:matter-labs/zk…
Apr 25, 2024
3093cde
Fix
Apr 25, 2024
fde3be1
Move prover_dal types to basic_types::prover_dal module
ilitteri Apr 25, 2024
efd70ac
Refactor query
ilitteri Apr 25, 2024
6ec0ac3
Merge pull request #1789 from matter-labs/prover_cli_compressor_status
ilitteri Apr 25, 2024
1830ced
add bwg query
ColoCarletti Apr 25, 2024
a6105b8
merge compresor job
ColoCarletti Apr 25, 2024
bbe5dc3
add status for prover jobs
ColoCarletti Apr 26, 2024
111884b
add title for prover jobs
ColoCarletti Apr 26, 2024
9fef23d
Refactor
ilitteri Apr 26, 2024
01ecd2f
add leaf query
ColoCarletti Apr 29, 2024
25e089b
Merge branch 'prover_cli_status_command' into prover_cli_status_comma…
ColoCarletti Apr 29, 2024
590c58b
fix merge
ColoCarletti Apr 29, 2024
e08b05e
add querys
ColoCarletti Apr 29, 2024
cf069b9
fix witnes jobs status
ColoCarletti Apr 30, 2024
6ad5566
rm comments
ColoCarletti Apr 30, 2024
a3f1938
Fix & refactors
ilitteri Apr 30, 2024
78c7cd8
zk lint rust
ilitteri Apr 30, 2024
33729b8
Fix import
ilitteri Apr 30, 2024
1efde33
Merge branch 'prover_cli_status_command' of github.com:matter-labs/zk…
ilitteri Apr 30, 2024
112ae93
fix import
ColoCarletti Apr 30, 2024
e1854fa
zk lint rust
ilitteri Apr 30, 2024
ccab3d3
Merge branch 'prover_cli_status_command_lwg' of github.com:matter-lab…
ColoCarletti Apr 30, 2024
fa26c26
fix wg task types
ColoCarletti Apr 30, 2024
dbaf3b4
Merge pull request #1830 from matter-labs/prover_cli_status_command_lwg
ColoCarletti Apr 30, 2024
4124f63
Merge main
ColoCarletti Apr 30, 2024
a4fde1f
update README
ColoCarletti Apr 30, 2024
10fdcb9
fix EIP Blob use
ColoCarletti Apr 30, 2024
8a1101f
fix typos
ColoCarletti Apr 30, 2024
c1282a5
Merge branch 'main' into prover_cli_status_command
ilitteri May 2, 2024
1bbde93
Fix typo
ilitteri May 2, 2024
16be778
Merge branch 'main' into prover_cli_status_command
ilitteri May 2, 2024
1810ae0
Merge branch 'main' into prover_cli_status_command
ilitteri May 2, 2024
6f45f43
update cargo.toml
ColoCarletti May 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions prover/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion prover/prover_cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,3 @@ zksync_types.workspace = true
zksync_prover_fri_types.workspace = true
zksync_prover_interface.workspace = true
prover_dal.workspace = true

ColoCarletti marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 3 additions & 1 deletion prover/prover_cli/src/cli.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
use clap::{command, Parser, Subcommand};

use crate::commands::get_file_info;
use crate::commands::{get_file_info, jobs_status};

pub const VERSION_STRING: &str = env!("CARGO_PKG_VERSION");

Expand All @@ -14,12 +14,14 @@ struct ProverCLI {
#[derive(Subcommand)]
enum ProverCommand {
FileInfo(get_file_info::Args),
StatusJobs(jobs_status::Args),
}

pub async fn start() -> anyhow::Result<()> {
let ProverCLI { command } = ProverCLI::parse();
match command {
ProverCommand::FileInfo(args) => get_file_info::run(args).await?,
ProverCommand::StatusJobs(args) => jobs_status::run(args).await?,
};

Ok(())
Expand Down
63 changes: 63 additions & 0 deletions prover/prover_cli/src/commands/jobs_status.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
use anyhow::Context as _;
use clap::Args as ClapArgs;
use prover_dal::{ConnectionPool, Prover, ProverDal};
use zksync_basic_types::{prover_dal::JobCountStatistics, L1BatchNumber};
use zksync_config::PostgresConfig;
use zksync_env_config::FromEnv;

#[derive(ClapArgs)]
pub(crate) struct Args {
#[clap(short, long, num_args = 0..)]
batch: Vec<L1BatchNumber>,
#[clap(short, long, default_value("false"))]
verbose: bool,
}

fn pretty_print_job_status(
l1_batch_number: &L1BatchNumber,
statistics: &JobCountStatistics,
verbose: bool,
) {
let total_jobs =
statistics.queued + statistics.in_progress + statistics.failed + statistics.successful;
let progress = (statistics.successful as f32 / total_jobs as f32) * 100.0;
println!("Batch number: {}", l1_batch_number);
println!(
"Progress: {:.2}% ({}/{})",
progress, statistics.successful, total_jobs
);
if verbose {
println!("In progress: {}", statistics.in_progress);
println!("Queued: {}", statistics.in_progress);
println!("Successful: {}", statistics.in_progress);
}
println!("Failed: {}", statistics.failed);
}

pub(crate) async fn run(args: Args) -> anyhow::Result<()> {
let postgres_config = PostgresConfig::from_env().context("PostgresConfig::from_env()")?;

let prover_connection_pool = ConnectionPool::<Prover>::builder(
postgres_config.prover_url()?,
postgres_config.max_connections()?,
)
.build()
.await
.context("failed to build a prover_connection_pool")?;

let mut conn = prover_connection_pool.connection().await.unwrap();
let stats = conn
.fri_prover_jobs_dal()
.get_prover_jobs_stats_for_batch(args.batch)
.await;
if stats.len() > 0 {
for row in &stats {
let (l1_batch_number, statistics) = row;

pretty_print_job_status(l1_batch_number, statistics, args.verbose)
}
} else {
println!("No batches found.")
}
Ok(())
}
1 change: 1 addition & 0 deletions prover/prover_cli/src/commands/mod.rs
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
pub(crate) mod get_file_info;
pub(crate) mod jobs_status;

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

53 changes: 53 additions & 0 deletions prover/prover_dal/src/fri_prover_dal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -594,4 +594,57 @@ impl FriProverDal<'_, '_> {
.unwrap()
.unwrap_or(0) as usize
}

pub async fn get_prover_jobs_stats_for_batch(
&mut self,
l1_batches_numbers: Vec<L1BatchNumber>,
) -> HashMap<L1BatchNumber, JobCountStatistics> {
{
sqlx::query!(
r#"
SELECT
COUNT(*) AS "count!",
l1_batch_number AS "l1_batch_number!",
status AS "status!"
FROM
prover_jobs_fri
WHERE
l1_batch_number = ANY ($1)
GROUP BY
l1_batch_number,
status
"#,
&l1_batches_numbers
.into_iter()
.map(|x| i64::from(x.0))
.collect::<Vec<i64>>()
)
.fetch_all(self.storage.conn())
.await
.unwrap()
.into_iter()
.map(|row| (row.l1_batch_number, row.status, row.count as usize))
.fold(
HashMap::new(),
|mut acc, (l1_batch_number, status, value)| {
let stats = acc.entry(L1BatchNumber(l1_batch_number as u32)).or_insert(
JobCountStatistics {
queued: 0,
in_progress: 0,
failed: 0,
successful: 0,
},
);
match status.as_ref() {
"queued" => stats.queued = value,
"in_progress" => stats.in_progress = value,
"failed" => stats.failed = value,
"successful" => stats.successful = value,
_ => (),
}
acc
},
)
}
}
}
Loading