Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[forge] Add junit support #15035

Merged
merged 1 commit into from
Oct 29, 2024
Merged

[forge] Add junit support #15035

merged 1 commit into from
Oct 29, 2024

Conversation

perryjrandall
Copy link
Contributor

@perryjrandall perryjrandall commented Oct 21, 2024

Add junit support for forge

This will allow us to onboard forge onto trunk flaky test detection

Also make suite choice more clean

Test Plan: running on a subsequent PR (due to pull request target)
https://github.com/aptos-labs/aptos-core/pull/15042/checks

also running on this PR (to ensure behavior doesnt change without flag)

Copy link

trunk-io bot commented Oct 21, 2024

⏱️ 7h 40m total CI duration on this PR
Slowest 15 Jobs Cumulative Duration Recent Runs
execution-performance / single-node-performance 3h 30m 🟥🟥🟥🟩🟩 (+5 more)
execution-performance / test-target-determinator 37m 🟩🟩🟩🟩🟩 (+5 more)
test-target-determinator 31m 🟩🟩🟩🟩 (+5 more)
check 23m 🟥🟩🟩🟩 (+4 more)
rust-cargo-deny 16m 🟩🟩🟩🟩 (+5 more)
fetch-last-released-docker-image-tag 15m 🟩🟩🟩🟩🟩 (+5 more)
check-dynamic-deps 13m 🟩🟩🟩🟩🟩 (+7 more)
rust-move-tests 10m 🟩
rust-move-tests 10m 🟩
rust-move-tests 10m
rust-move-tests 9m 🟩
rust-move-tests 9m 🟩
rust-move-tests 9m 🟩
rust-doc-tests 5m 🟩
rust-doc-tests 5m 🟩

settingsfeedbackdocs ⋅ learn more about trunk.io

@perryjrandall perryjrandall requested a review from a team as a code owner October 22, 2024 06:33
@perryjrandall perryjrandall added CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR CICD:run-framework-upgrade-test labels Oct 22, 2024
@perryjrandall perryjrandall requested review from ibalajiarun, banool, aluon and a team October 22, 2024 06:42
.github/workflows/workflow-run-forge.yaml Outdated Show resolved Hide resolved
testsuite/forge-cli/src/main.rs Show resolved Hide resolved
testsuite/forge/src/config.rs Show resolved Hide resolved
testsuite/forge/src/result.rs Outdated Show resolved Hide resolved
testsuite/forge/src/result.rs Outdated Show resolved Hide resolved
@perryjrandall perryjrandall force-pushed the perry/add-junit-forge branch 4 times, most recently from 2a0bcd5 to a9ebd4b Compare October 22, 2024 18:52
@perryjrandall perryjrandall force-pushed the perry/add-junit-forge branch 3 times, most recently from 8e36106 to 307fc5d Compare October 24, 2024 01:48

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Add junit support for forge

This will allow us to onboard forge onto trunk flaky test detection

Also make suite choice more clean

Test Plan: running on PR

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor

✅ Forge suite realistic_env_max_load success on e1e30413f09e9a6a6867866f85e8f4796f4421e2

two traffics test: inner traffic : committed: 14096.60 txn/s, latency: 2817.33 ms, (p50: 2700 ms, p70: 2700, p90: 2900 ms, p99: 11500 ms), latency samples: 5360100
two traffics test : committed: 99.92 txn/s, latency: 1639.48 ms, (p50: 1400 ms, p70: 1400, p90: 1600 ms, p99: 7900 ms), latency samples: 1740
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.021, avg: 1.622", "ConsensusProposalToOrdered: max: 0.316, avg: 0.293", "ConsensusOrderedToCommit: max: 0.368, avg: 0.360", "ConsensusProposalToCommit: max: 0.663, avg: 0.654"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.88s no progress at version 5122150 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.43s no progress at version 5122148 (avg 7.98s) [limit 15].
Test Ok

Copy link
Contributor

✅ Forge suite framework_upgrade success on 9c922ebe94f5ff4b58df4617f3ff003e2ce10ccd ==> e1e30413f09e9a6a6867866f85e8f4796f4421e2

Compatibility test results for 9c922ebe94f5ff4b58df4617f3ff003e2ce10ccd ==> e1e30413f09e9a6a6867866f85e8f4796f4421e2 (PR)
Upgrade the nodes to version: e1e30413f09e9a6a6867866f85e8f4796f4421e2
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1253.68 txn/s, submitted: 1256.63 txn/s, failed submission: 2.95 txn/s, expired: 2.95 txn/s, latency: 2481.67 ms, (p50: 2400 ms, p70: 2700, p90: 3300 ms, p99: 4600 ms), latency samples: 110340
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1446.66 txn/s, submitted: 1449.56 txn/s, failed submission: 2.89 txn/s, expired: 2.89 txn/s, latency: 2094.77 ms, (p50: 1800 ms, p70: 2300, p90: 3000 ms, p99: 4200 ms), latency samples: 130000
5. check swarm health
Compatibility test for 9c922ebe94f5ff4b58df4617f3ff003e2ce10ccd ==> e1e30413f09e9a6a6867866f85e8f4796f4421e2 passed
Upgrade the remaining nodes to version: e1e30413f09e9a6a6867866f85e8f4796f4421e2
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1403.00 txn/s, submitted: 1405.47 txn/s, failed submission: 2.47 txn/s, expired: 2.47 txn/s, latency: 2181.02 ms, (p50: 2100 ms, p70: 2400, p90: 2900 ms, p99: 4300 ms), latency samples: 124900
Test Ok

Copy link
Contributor

✅ Forge suite compat success on 9c922ebe94f5ff4b58df4617f3ff003e2ce10ccd ==> e1e30413f09e9a6a6867866f85e8f4796f4421e2

Compatibility test results for 9c922ebe94f5ff4b58df4617f3ff003e2ce10ccd ==> e1e30413f09e9a6a6867866f85e8f4796f4421e2 (PR)
1. Check liveness of validators at old version: 9c922ebe94f5ff4b58df4617f3ff003e2ce10ccd
compatibility::simple-validator-upgrade::liveness-check : committed: 15073.30 txn/s, latency: 2251.20 ms, (p50: 2100 ms, p70: 2200, p90: 2500 ms, p99: 6000 ms), latency samples: 485920
2. Upgrading first Validator to new version: e1e30413f09e9a6a6867866f85e8f4796f4421e2
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6081.73 txn/s, latency: 4731.63 ms, (p50: 5400 ms, p70: 5800, p90: 5900 ms, p99: 5900 ms), latency samples: 112560
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 5934.13 txn/s, latency: 5459.95 ms, (p50: 6000 ms, p70: 6100, p90: 6600 ms, p99: 7400 ms), latency samples: 203380
3. Upgrading rest of first batch to new version: e1e30413f09e9a6a6867866f85e8f4796f4421e2
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 5856.04 txn/s, latency: 4959.34 ms, (p50: 5500 ms, p70: 6000, p90: 6200 ms, p99: 6300 ms), latency samples: 112140
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5335.84 txn/s, latency: 5809.80 ms, (p50: 6200 ms, p70: 6300, p90: 6600 ms, p99: 7800 ms), latency samples: 200900
4. upgrading second batch to new version: e1e30413f09e9a6a6867866f85e8f4796f4421e2
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 8550.12 txn/s, latency: 3305.56 ms, (p50: 3400 ms, p70: 3700, p90: 4800 ms, p99: 5100 ms), latency samples: 158080
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8082.62 txn/s, latency: 3956.28 ms, (p50: 3800 ms, p70: 4000, p90: 6300 ms, p99: 6800 ms), latency samples: 300120
5. check swarm health
Compatibility test for 9c922ebe94f5ff4b58df4617f3ff003e2ce10ccd ==> e1e30413f09e9a6a6867866f85e8f4796f4421e2 passed
Test Ok

@perryjrandall perryjrandall merged commit 82bd006 into main Oct 29, 2024
48 checks passed
@perryjrandall perryjrandall deleted the perry/add-junit-forge branch October 29, 2024 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR CICD:run-framework-upgrade-test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants