[CLI] Add a CLI command to install prover dependencies #14683

rahxephon89 · 2024-09-18T22:16:48Z

Description

This PR adds a new command aptos update prover-dependencies to install the prover dependency: z3, boogie and cvc5 and sets the required environment variables. With this new command, developers don't need to clone aptos-core and run dev_setup.sh to install prover dependencies. Once this feature is released, we will update the prover installation guide.

The binaries of the prover dependency is maintained in the repo prover-dependency.

Note that CI will still use dev_setup.sh to install prover dependencies and the check is added to the command
to guarantee versions installed by the CLI is compatible with the requirement for the prover.

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

Manually tested the command on macos, linux and windows.

Key Areas to Review

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-09-18T22:16:51Z

⏱️ 5h 55m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
rust-move-unit-coverage	22m	🟩
general-lints	22m	🟩 🟩 ⬜ ⬜ 🟩 (+8 more)
rust-cargo-deny	21m	🟩 🟩 ⬜ 🟩 🟩 (+8 more)
run-tests-main-branch	20m	🟥 🟥 🟥
rust-move-unit-coverage	17m	🟩
check-dynamic-deps	16m	🟩 🟩 🟩 🟩 🟩 (+8 more)
rust-move-unit-coverage	15m	🟩
rust-move-unit-coverage	15m	🟩
rust-move-unit-coverage	15m	🟩
rust-move-unit-coverage	15m	🟩
rust-images / rust-all	14m	🟩
rust-move-unit-coverage	11m	⬜
rust-move-tests	10m	🟩
rust-move-unit-coverage	10m	🟩
rust-move-unit-coverage	10m	🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

rahxephon89 · 2024-09-18T22:17:03Z

[CLI] Add a CLI command to install prover dependencies #14683 👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @rahxephon89 and the rest of your teammates on Graphite

codecov · 2024-09-18T22:32:44Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.8%. Comparing base (bbe2ba8) to head (ecac2fe).
Report is 8 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #14683   +/-   ##
=======================================
  Coverage    59.8%    59.8%           
=======================================
  Files         853      853           
  Lines      207917   207923    +6     
=======================================
+ Hits       124342   124360   +18     
+ Misses      83575    83563   -12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

rahxephon89 · 2024-09-18T23:51:46Z

[CLI] Add a CLI command to install prover dependencies #14683 👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @rahxephon89 and the rest of your teammates on Graphite

crates/aptos/src/update/tool.rs

crates/aptos/src/update/prover_dependency.rs

rahxephon89 · 2024-09-25T19:54:46Z

crates/aptos/src/update/movefmt.rs

@@ -13,7 +13,7 @@ use self_update::update::ReleaseUpdate;
 use std::path::PathBuf;

 const FORMATTER_BINARY_NAME: &str = "movefmt";
-const TARGET_FORMATTER_VERSION: &str = "1.0.4";
+const TARGET_FORMATTER_VERSION: &str = "1.0.5";


piggyback the latest version of movefmt

banool · 2024-09-26T11:26:13Z

crates/aptos/src/update/prover_dependencies.rs

+            install_path.to_string_lossy()
+        );
+        if env::var(env_var).is_err() {
+            println!("Please use the `source` command or reboot the terminal to check whether {} is set with the correct value. \


Suggested change

println!("Please use the `source` command or reboot the terminal to check whether {} is set with the correct value. \

eprintln!("Please use the `source` command or reboot the terminal to check whether {} is set with the correct value. \

banool · 2024-09-26T11:27:18Z

crates/aptos/src/update/prover_dependencies.rs

+        set_env::set(env_var, install_path.to_string_lossy())
+            .map_err(|e| CliError::UnexpectedError(format!("Failed to set {}: {}", env_var, e)))?;
+        println!(
+            "Added {} to environment with value: {}.",


This message should make it clear that we actually added this to their profile, so it is added to the environment permanently.

banool · 2024-09-26T11:28:08Z

crates/aptos/src/update/prover_dependencies.rs

+            .await?;
+        println!("{}", res);
+
+        #[cfg(unix)]


So I guess Windows doesn't need this tool?

Yeah, it is not supported windows in the current version, which may change in the future.

banool · 2024-09-26T11:28:37Z

crates/aptos/src/update/prover_dependency_installer.rs

+    pub binary_name: String,
+
+    pub exe_name: String,
+
+    pub env_var: String,
+
+    pub version_match_string: String,
+
+    pub version_option_string: String,
+
+    pub target_version: String,
+
+    pub install_dir: Option<PathBuf>,
+
+    pub check: bool,


Some comments on these would be nice, it's not clear what they do.

banool · 2024-09-26T11:29:00Z

crates/aptos/src/update/prover_dependency_installer.rs

+
+impl BinaryUpdater for DependencyInstaller {
+    fn check(&self) -> bool {
+        false


Shouldn't this use self.check?

banool · 2024-09-26T11:30:00Z

crates/aptos/src/update/update_helper.rs

@@ -63,6 +66,12 @@ pub fn build_updater(
        None => "0.0.0",
    };

+    println!(
+        "install_dir:{:?}, is directory:{}",


Suggested change

"install_dir:{:?}, is directory:{}",

"install_dir: {:?}, is directory: {}",

removed the print statement

banool · 2024-09-26T11:30:13Z

crates/aptos/src/update/update_helper.rs

@@ -48,7 +48,10 @@ pub fn build_updater(
    let target = format!("{}-{}", arch_str, target);

    let install_dir = match install_dir.clone() {
-        Some(dir) => dir,
+        Some(dir) => {
+            println!("dir:{:?}", dir);


Suggested change

println!("dir:{:?}", dir);

println!("dir: {:?}", dir);

banool · 2024-09-26T11:30:53Z

crates/aptos/src/update/update_helper.rs

@@ -48,7 +48,10 @@ pub fn build_updater(
    let target = format!("{}-{}", arch_str, target);

    let install_dir = match install_dir.clone() {
-        Some(dir) => dir,
+        Some(dir) => {


I suggested formatting improvements but these sort of look like debug prints, is it intentional that this and the one for install dir are left in the code?

yeah, it is for debugging, removed the print statement

banool

Looks good to me, thanks for getting it to this state!

github-actions · 2024-10-02T00:18:30Z

✅ Forge suite `compat` success on `7ef01a26f8d8a38610e3d364b722df517c970749` ==> `ecac2feab055a45b53ff4dc16aff82a05d0e37ce`

Compatibility test results for 7ef01a26f8d8a38610e3d364b722df517c970749 ==> ecac2feab055a45b53ff4dc16aff82a05d0e37ce (PR)
1. Check liveness of validators at old version: 7ef01a26f8d8a38610e3d364b722df517c970749
compatibility::simple-validator-upgrade::liveness-check : committed: 10408.75 txn/s, latency: 2772.36 ms, (p50: 2100 ms, p70: 2400, p90: 2700 ms, p99: 24000 ms), latency samples: 414220
2. Upgrading first Validator to new version: ecac2feab055a45b53ff4dc16aff82a05d0e37ce
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6652.72 txn/s, latency: 4245.09 ms, (p50: 4800 ms, p70: 5100, p90: 5300 ms, p99: 5400 ms), latency samples: 123880
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6282.08 txn/s, latency: 4994.91 ms, (p50: 4900 ms, p70: 5100, p90: 7500 ms, p99: 7900 ms), latency samples: 214640
3. Upgrading rest of first batch to new version: ecac2feab055a45b53ff4dc16aff82a05d0e37ce
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7102.44 txn/s, latency: 3780.91 ms, (p50: 3700 ms, p70: 4700, p90: 5500 ms, p99: 5600 ms), latency samples: 130940
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7081.46 txn/s, latency: 4506.67 ms, (p50: 4500 ms, p70: 4800, p90: 6600 ms, p99: 6900 ms), latency samples: 238280
4. upgrading second batch to new version: ecac2feab055a45b53ff4dc16aff82a05d0e37ce
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10235.96 txn/s, latency: 2573.54 ms, (p50: 2700 ms, p70: 2900, p90: 3200 ms, p99: 3400 ms), latency samples: 190020
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 9818.36 txn/s, latency: 3199.54 ms, (p50: 3200 ms, p70: 3300, p90: 3500 ms, p99: 3700 ms), latency samples: 325520
5. check swarm health
Compatibility test for 7ef01a26f8d8a38610e3d364b722df517c970749 ==> ecac2feab055a45b53ff4dc16aff82a05d0e37ce passed
Test Ok

github-actions · 2024-10-02T00:19:16Z

✅ Forge suite `framework_upgrade` success on `7ef01a26f8d8a38610e3d364b722df517c970749` ==> `ecac2feab055a45b53ff4dc16aff82a05d0e37ce`

Compatibility test results for 7ef01a26f8d8a38610e3d364b722df517c970749 ==> ecac2feab055a45b53ff4dc16aff82a05d0e37ce (PR)
Upgrade the nodes to version: ecac2feab055a45b53ff4dc16aff82a05d0e37ce
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1300.85 txn/s, submitted: 1302.62 txn/s, failed submission: 1.77 txn/s, expired: 1.77 txn/s, latency: 2705.82 ms, (p50: 2400 ms, p70: 2700, p90: 4900 ms, p99: 6300 ms), latency samples: 102860
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1109.37 txn/s, submitted: 1111.39 txn/s, failed submission: 2.02 txn/s, expired: 2.02 txn/s, latency: 2731.57 ms, (p50: 2400 ms, p70: 2800, p90: 4800 ms, p99: 6000 ms), latency samples: 98920
5. check swarm health
Compatibility test for 7ef01a26f8d8a38610e3d364b722df517c970749 ==> ecac2feab055a45b53ff4dc16aff82a05d0e37ce passed
Upgrade the remaining nodes to version: ecac2feab055a45b53ff4dc16aff82a05d0e37ce
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1182.69 txn/s, submitted: 1184.72 txn/s, failed submission: 2.03 txn/s, expired: 2.03 txn/s, latency: 2813.29 ms, (p50: 2400 ms, p70: 2700, p90: 4800 ms, p99: 6300 ms), latency samples: 104740
Test Ok

github-actions · 2024-10-02T00:19:20Z

✅ Forge suite `realistic_env_max_load` success on `ecac2feab055a45b53ff4dc16aff82a05d0e37ce`

two traffics test: inner traffic : committed: 12133.24 txn/s, submitted: 12148.86 txn/s, expired: 15.62 txn/s, latency: 3181.95 ms, (p50: 3000 ms, p70: 3000, p90: 3300 ms, p99: 12100 ms), latency samples: 4613360
two traffics test : committed: 100.07 txn/s, latency: 1620.20 ms, (p50: 1500 ms, p70: 1600, p90: 1800 ms, p99: 8600 ms), latency samples: 2000
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.277, avg: 0.233", "QsPosToProposal: max: 1.281, avg: 1.170", "ConsensusProposalToOrdered: max: 0.346, avg: 0.329", "ConsensusOrderedToCommit: max: 0.720, avg: 0.612", "ConsensusProposalToCommit: max: 1.044, avg: 0.941"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.28s no progress at version 32909 (avg 0.23s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 7.72s no progress at version 2366615 (avg 4.36s) [limit 15].
Test Ok

rahxephon89 changed the title ~~add prover dependency~~ [WIP][CLI] add a CLI command to download prover dependencies Sep 18, 2024

rahxephon89 force-pushed the teng/prover-dependency branch from b0cb61e to 4961e5a Compare September 18, 2024 23:51

rahxephon89 force-pushed the teng/prover-dependency branch 2 times, most recently from 838ae5b to 6bc3ba7 Compare September 20, 2024 02:44

rahxephon89 changed the title ~~[WIP][CLI] add a CLI command to download prover dependencies~~ [CLI] add a CLI command to download prover dependencies Sep 20, 2024

rahxephon89 marked this pull request as ready for review September 20, 2024 05:25

rahxephon89 requested review from gregnazario and banool as code owners September 20, 2024 05:25

rahxephon89 requested a review from wrwg September 20, 2024 05:26

rahxephon89 changed the title ~~[CLI] add a CLI command to download prover dependencies~~ [CLI] add a CLI command to install prover dependencies Sep 20, 2024

rahxephon89 changed the title ~~[CLI] add a CLI command to install prover dependencies~~ [CLI] Add a CLI command to install prover dependencies Sep 20, 2024

rahxephon89 force-pushed the teng/prover-dependency branch from 6bc3ba7 to 7f7b09e Compare September 20, 2024 08:03

banool requested changes Sep 20, 2024

View reviewed changes

rahxephon89 marked this pull request as draft September 20, 2024 15:41

rahxephon89 force-pushed the teng/prover-dependency branch 4 times, most recently from 3e0770b to 938f1c0 Compare September 21, 2024 06:22

rahxephon89 marked this pull request as ready for review September 21, 2024 06:25

rahxephon89 requested a review from banool September 21, 2024 06:26

rahxephon89 force-pushed the teng/prover-dependency branch from 938f1c0 to 7054457 Compare September 21, 2024 18:08

rahxephon89 mentioned this pull request Sep 23, 2024

[Feature Request] Homebrew repo for Mac Arm64 versions of needed tools for Move prover and tests #9478

Open

rahxephon89 force-pushed the teng/prover-dependency branch from 7054457 to 9749770 Compare September 25, 2024 19:53

rahxephon89 commented Sep 25, 2024

View reviewed changes

rahxephon89 force-pushed the teng/prover-dependency branch from de38ab9 to 2f4f445 Compare September 25, 2024 19:55

banool reviewed Sep 26, 2024

View reviewed changes

rahxephon89 force-pushed the teng/prover-dependency branch 2 times, most recently from 1ece55d to b017c0a Compare September 26, 2024 17:30

rahxephon89 requested a review from banool September 26, 2024 17:31

rahxephon89 force-pushed the teng/prover-dependency branch from b017c0a to 2880eb0 Compare September 26, 2024 22:07

rahxephon89 requested a review from junkil-park September 27, 2024 18:06

rahxephon89 added 2 commits October 1, 2024 10:20

add prover dependency

38710e4

update fmt version

225aa1c

rahxephon89 force-pushed the teng/prover-dependency branch from 2880eb0 to 225aa1c Compare October 1, 2024 17:22

banool approved these changes Oct 1, 2024

View reviewed changes

gregnazario approved these changes Oct 1, 2024

View reviewed changes

Merge branch 'main' into teng/prover-dependency

ecac2fe

rahxephon89 enabled auto-merge (squash) October 1, 2024 22:13

This comment has been minimized.

Sign in to view

rahxephon89 merged commit 5a5d456 into main Oct 2, 2024
90 of 103 checks passed

rahxephon89 deleted the teng/prover-dependency branch October 2, 2024 00:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CLI] Add a CLI command to install prover dependencies #14683

[CLI] Add a CLI command to install prover dependencies #14683

rahxephon89 commented Sep 18, 2024 •

edited

Loading

trunk-io bot commented Sep 18, 2024 •

edited

Loading

rahxephon89 commented Sep 18, 2024

codecov bot commented Sep 18, 2024 •

edited

Loading

rahxephon89 commented Sep 18, 2024

rahxephon89 Sep 25, 2024

banool Sep 26, 2024

rahxephon89 Sep 26, 2024

banool Sep 26, 2024

rahxephon89 Sep 26, 2024

banool Sep 26, 2024

rahxephon89 Sep 26, 2024

banool Sep 26, 2024

rahxephon89 Sep 26, 2024

banool Sep 26, 2024

rahxephon89 Sep 26, 2024

banool Sep 26, 2024

rahxephon89 Sep 26, 2024

banool Sep 26, 2024

banool Sep 26, 2024

rahxephon89 Sep 26, 2024

banool left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

	println!("Please use the `source` command or reboot the terminal to check whether {} is set with the correct value. \
	eprintln!("Please use the `source` command or reboot the terminal to check whether {} is set with the correct value. \

	"install_dir:{:?}, is directory:{}",
	"install_dir: {:?}, is directory: {}",

[CLI] Add a CLI command to install prover dependencies #14683

[CLI] Add a CLI command to install prover dependencies #14683

Conversation

rahxephon89 commented Sep 18, 2024 • edited Loading

Description

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

Key Areas to Review

Checklist

trunk-io bot commented Sep 18, 2024 • edited Loading

rahxephon89 commented Sep 18, 2024

codecov bot commented Sep 18, 2024 • edited Loading

Codecov Report

rahxephon89 commented Sep 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banool left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Oct 2, 2024

✅ Forge suite compat success on 7ef01a26f8d8a38610e3d364b722df517c970749 ==> ecac2feab055a45b53ff4dc16aff82a05d0e37ce

github-actions bot commented Oct 2, 2024

✅ Forge suite framework_upgrade success on 7ef01a26f8d8a38610e3d364b722df517c970749 ==> ecac2feab055a45b53ff4dc16aff82a05d0e37ce

github-actions bot commented Oct 2, 2024

✅ Forge suite realistic_env_max_load success on ecac2feab055a45b53ff4dc16aff82a05d0e37ce

rahxephon89 commented Sep 18, 2024 •

edited

Loading

trunk-io bot commented Sep 18, 2024 •

edited

Loading

codecov bot commented Sep 18, 2024 •

edited

Loading

✅ Forge suite `compat` success on `7ef01a26f8d8a38610e3d364b722df517c970749` ==> `ecac2feab055a45b53ff4dc16aff82a05d0e37ce`

✅ Forge suite `framework_upgrade` success on `7ef01a26f8d8a38610e3d364b722df517c970749` ==> `ecac2feab055a45b53ff4dc16aff82a05d0e37ce`

✅ Forge suite `realistic_env_max_load` success on `ecac2feab055a45b53ff4dc16aff82a05d0e37ce`