Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[forge] Split out test suites from main #15033

Merged
merged 1 commit into from
Oct 21, 2024
Merged

Conversation

perryjrandall
Copy link
Contributor

There are a lot of tests and cruft in with the main forge cli

Split them out into suites

Test Plan:

running forge on PR succeeds

Copy link

trunk-io bot commented Oct 21, 2024

⏱️ 1h 32m total CI duration on this PR
Slowest 15 Jobs Cumulative Duration Recent Runs
execution-performance / test-target-determinator 17m 🟩🟩🟩🟩
test-target-determinator 15m 🟩🟩🟩
rust-cargo-deny 9m 🟩🟩🟩🟩
check 8m 🟥🟥🟩🟩
fetch-last-released-docker-image-tag 6m 🟩🟩🟩🟩
rust-doc-tests 5m 🟩
rust-doc-tests 5m 🟩
rust-doc-tests 5m 🟩
check-dynamic-deps 4m 🟩🟩🟩🟩🟩
general-lints 2m 🟩🟩🟩🟩🟩
rust-move-tests 2m 🟩
rust-doc-tests 2m
semgrep/ci 2m 🟩🟩🟩🟩🟩
rust-move-tests 2m 🟩
rust-move-tests 2m 🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job Duration vs 7d avg Delta
execution-performance / single-node-performance 10s 20m -99%

settingsfeedbackdocs ⋅ learn more about trunk.io

@perryjrandall perryjrandall requested review from ibalajiarun, aluon and a team October 21, 2024 16:11
@perryjrandall perryjrandall enabled auto-merge (rebase) October 21, 2024 16:15

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

@perryjrandall perryjrandall force-pushed the perry/split-forge-suites branch from 237fba6 to 339f2cd Compare October 21, 2024 16:42

This comment has been minimized.

@perryjrandall perryjrandall force-pushed the perry/split-forge-suites branch from 339f2cd to 2e80bb3 Compare October 21, 2024 16:45

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

There are a lot of tests and cruft in with the main forge cli

Split them out into suites

Test Plan:

running forge on PR succeeds
@perryjrandall perryjrandall force-pushed the perry/split-forge-suites branch from 2e80bb3 to 5b2c544 Compare October 21, 2024 17:09

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor

✅ Forge suite realistic_env_max_load success on 5b2c5449a85d739859e682fecd2a6f6ef06696a6

two traffics test: inner traffic : committed: 12627.07 txn/s, latency: 3133.43 ms, (p50: 2700 ms, p70: 3000, p90: 3600 ms, p99: 10900 ms), latency samples: 4801220
two traffics test : committed: 100.02 txn/s, latency: 1865.55 ms, (p50: 1600 ms, p70: 1700, p90: 2100 ms, p99: 9000 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.241, avg: 0.223", "QsPosToProposal: max: 1.025, avg: 0.959", "ConsensusProposalToOrdered: max: 0.333, avg: 0.322", "ConsensusOrderedToCommit: max: 0.947, avg: 0.676", "ConsensusProposalToCommit: max: 1.267, avg: 0.998"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.22s no progress at version 2179610 (avg 0.23s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 7.54s no progress at version 2179608 (avg 7.54s) [limit 15].
Test Ok

Copy link
Contributor

✅ Forge suite compat success on b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 5b2c5449a85d739859e682fecd2a6f6ef06696a6

Compatibility test results for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 5b2c5449a85d739859e682fecd2a6f6ef06696a6 (PR)
1. Check liveness of validators at old version: b29f09f57e898d8d211c8bc3e303f6e50bba2266
compatibility::simple-validator-upgrade::liveness-check : committed: 12468.44 txn/s, latency: 2728.94 ms, (p50: 1900 ms, p70: 2100, p90: 3700 ms, p99: 22400 ms), latency samples: 470420
2. Upgrading first Validator to new version: 5b2c5449a85d739859e682fecd2a6f6ef06696a6
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 5162.67 txn/s, latency: 5588.13 ms, (p50: 6200 ms, p70: 6800, p90: 6900 ms, p99: 7000 ms), latency samples: 100800
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 4914.17 txn/s, latency: 6633.93 ms, (p50: 7400 ms, p70: 7600, p90: 7900 ms, p99: 8000 ms), latency samples: 170980
3. Upgrading rest of first batch to new version: 5b2c5449a85d739859e682fecd2a6f6ef06696a6
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 5442.48 txn/s, latency: 5310.39 ms, (p50: 6000 ms, p70: 6300, p90: 6500 ms, p99: 6600 ms), latency samples: 102740
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 4940.41 txn/s, latency: 6532.54 ms, (p50: 7000 ms, p70: 7100, p90: 8100 ms, p99: 9000 ms), latency samples: 166760
4. upgrading second batch to new version: 5b2c5449a85d739859e682fecd2a6f6ef06696a6
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 8323.71 txn/s, latency: 3213.77 ms, (p50: 3300 ms, p70: 3500, p90: 4000 ms, p99: 4600 ms), latency samples: 155280
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 7235.71 txn/s, latency: 4171.62 ms, (p50: 4000 ms, p70: 4500, p90: 6400 ms, p99: 8000 ms), latency samples: 244460
5. check swarm health
Compatibility test for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 5b2c5449a85d739859e682fecd2a6f6ef06696a6 passed
Test Ok

Copy link
Contributor

✅ Forge suite framework_upgrade success on b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 5b2c5449a85d739859e682fecd2a6f6ef06696a6

Compatibility test results for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 5b2c5449a85d739859e682fecd2a6f6ef06696a6 (PR)
Upgrade the nodes to version: 5b2c5449a85d739859e682fecd2a6f6ef06696a6
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1001.03 txn/s, submitted: 1003.30 txn/s, failed submission: 2.27 txn/s, expired: 2.27 txn/s, latency: 3133.00 ms, (p50: 2700 ms, p70: 3500, p90: 5500 ms, p99: 7700 ms), latency samples: 88180
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 859.04 txn/s, submitted: 860.36 txn/s, failed submission: 1.32 txn/s, expired: 1.32 txn/s, latency: 3484.12 ms, (p50: 3000 ms, p70: 4100, p90: 5900 ms, p99: 8100 ms), latency samples: 78120
5. check swarm health
Compatibility test for b29f09f57e898d8d211c8bc3e303f6e50bba2266 ==> 5b2c5449a85d739859e682fecd2a6f6ef06696a6 passed
Upgrade the remaining nodes to version: 5b2c5449a85d739859e682fecd2a6f6ef06696a6
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 873.47 txn/s, submitted: 875.46 txn/s, failed submission: 1.99 txn/s, expired: 1.99 txn/s, latency: 3484.11 ms, (p50: 2900 ms, p70: 4200, p90: 6100 ms, p99: 7400 ms), latency samples: 79040
Test Ok

Copy link
Contributor

@aluon aluon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -617,7 +518,7 @@ fn get_test_suite(

// Otherwise, check the test name against the grouped test suites
if let Some(test_suite) = get_land_blocking_test(test_name, duration, test_cmd) {
return Ok(test_suite);
Ok(test_suite)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also remove the return statements from the other branches to match this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woops yeah i already fixed this in another commit forgot to pick

// SPDX-License-Identifier: Apache-2.0

pub mod dag;
pub mod db;
Copy link
Contributor

@aluon aluon Oct 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably for a future PR. Now that we have the test suites split up into different modules, it'd be great to have module-level docs for what each test suite is doing. @igor-aptos added some comments for forge-stable in #14775 , maybe we should move some of the docs to these files?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah in general i think these should live with the tests themselves not in docs on a forge workflow

@perryjrandall perryjrandall merged commit fbf09ce into main Oct 21, 2024
48 checks passed
@perryjrandall perryjrandall deleted the perry/split-forge-suites branch October 21, 2024 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants