Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Forge continuous tests - tune params based on recent runs #8803

Merged
merged 1 commit into from
Jun 26, 2023
Merged

Forge continuous tests - tune params based on recent runs #8803

merged 1 commit into from
Jun 26, 2023

Conversation

igor-aptos
Copy link
Contributor

  • increase cpu on high throughput
  • increase p99 latency for sweep, for variance, but add p90 test with lower values
  • reduce duration of long test, until long test can run successfully

Description

Test Plan

@igor-aptos igor-aptos requested a review from a team as a code owner June 22, 2023 22:04
bchocho
bchocho approved these changes Jun 22, 2023
View reviewed changes
Copy link
Contributor

@bchocho bchocho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

igor-aptos
igor-aptos commented Jun 25, 2023
View reviewed changes
.github/workflows/forge-stable.yaml Outdated
Comment on lines 122 to 123
# reduce temporarily, until longer tests can pass.
FORGE_RUNNER_DURATION_SECS: 2400
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# reduce temporarily, until longer tests can pass.
FORGE_RUNNER_DURATION_SECS: 2400
FORGE_RUNNER_DURATION_SECS: 7200

@igor-aptos igor-aptos force-pushed the igor/tune_continuous branch from 2541900 to 5043d68 Compare June 26, 2023 16:52
JoshLind
JoshLind approved these changes Jun 26, 2023
View reviewed changes
Copy link
Contributor

@JoshLind JoshLind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unblocking 😄

testsuite/forge-cli/src/main.rs Outdated
// Check that we don't use more than 12 CPU cores for 30% of the time.
MetricsThreshold::new(12, 30),
// Tuned for throughput uses more cores than regular tests,
// as it achieves higher throughput
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: fullstop? 😄

@igor-aptos igor-aptos force-pushed the igor/tune_continuous branch from 5043d68 to 4a03276 Compare June 26, 2023 18:22
@igor-aptos igor-aptos enabled auto-merge (squash) June 26, 2023 18:22
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@github-actions
Copy link
Contributor

✅ Forge suite compat success on aptos-node-v1.4.4 ==> 4a0327620ef5808dda5d8722c28c6bf72f95eafa

Compatibility test results for aptos-node-v1.4.4 ==> 4a0327620ef5808dda5d8722c28c6bf72f95eafa (PR)
1. Check liveness of validators at old version: aptos-node-v1.4.4
compatibility::simple-validator-upgrade::liveness-check : committed: 6888 txn/s, latency: 4748 ms, (p50: 4700 ms, p90: 6300 ms, p99: 8300 ms), latency samples: 241100
2. Upgrading first Validator to new version: 4a0327620ef5808dda5d8722c28c6bf72f95eafa
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 4314 txn/s, latency: 7429 ms, (p50: 8300 ms, p90: 9600 ms, p99: 10200 ms), latency samples: 163940
3. Upgrading rest of first batch to new version: 4a0327620ef5808dda5d8722c28c6bf72f95eafa
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 3925 txn/s, latency: 8039 ms, (p50: 8800 ms, p90: 10100 ms, p99: 10300 ms), latency samples: 149160
4. upgrading second batch to new version: 4a0327620ef5808dda5d8722c28c6bf72f95eafa
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 6516 txn/s, latency: 5072 ms, (p50: 5200 ms, p90: 6900 ms, p99: 8200 ms), latency samples: 228080
5. check swarm health
Compatibility test for aptos-node-v1.4.4 ==> 4a0327620ef5808dda5d8722c28c6bf72f95eafa passed
Test Ok

@github-actions
Copy link
Contributor

✅ Forge suite land_blocking success on 4a0327620ef5808dda5d8722c28c6bf72f95eafa

performance benchmark : committed: 5622 txn/s, submitted: 5623 txn/s, latency: 7088 ms, (p50: 6000 ms, p90: 9700 ms, p99: 25600 ms), latency samples: 2400980
Max round gap was 1 [limit 4] at version 509580. Max no progress secs was 3.72676 [limit 10] at version 1517637.
Test Ok

@github-actions
Copy link
Contributor

✅ Forge suite framework_upgrade success on aptos-node-v1.4.4 ==> 4a0327620ef5808dda5d8722c28c6bf72f95eafa

Compatibility test results for aptos-node-v1.4.4 ==> 4a0327620ef5808dda5d8722c28c6bf72f95eafa (PR)
Upgrade the nodes to version: 4a0327620ef5808dda5d8722c28c6bf72f95eafa
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 4353 txn/s, latency: 7321 ms, (p50: 7800 ms, p90: 9300 ms, p99: 14000 ms), latency samples: 165420
5. check swarm health
Compatibility test for aptos-node-v1.4.4 ==> 4a0327620ef5808dda5d8722c28c6bf72f95eafa passed
Test Ok

@igor-aptos igor-aptos merged commit 3494597 into main Jun 26, 2023
@igor-aptos igor-aptos deleted the igor/tune_continuous branch June 26, 2023 19:00
gedigi pushed a commit that referenced this pull request Aug 2, 2023
@igor-aptos @gedigi
Forge continuous tests - tune params based on recent runs (#8803)
b2dea67
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rustielin rustielin rustielin approved these changes

@JoshLind JoshLind JoshLind approved these changes

@bchocho bchocho bchocho approved these changes

@sionescu sionescu Awaiting requested review from sionescu

@ibalajiarun ibalajiarun Awaiting requested review from ibalajiarun

@perryjrandall perryjrandall Awaiting requested review from perryjrandall

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@igor-aptos @bchocho @JoshLind @rustielin