Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[consensus] record timeout author with reason in metrics #15157

Merged
merged 1 commit into from
Nov 1, 2024

Conversation

ibalajiarun
Copy link
Contributor

Description

Record the failed author as label in the AGGREGATED_ROUND_TIMEOUT_REASON metric, so we can identify which author failed for what reason.

Copy link

trunk-io bot commented Nov 1, 2024

⏱️ 38m total CI duration on this PR
Slowest 15 Jobs Cumulative Duration Recent Runs
rust-cargo-deny 5m 🟩🟩🟩
execution-performance / test-target-determinator 5m 🟩
rust-doc-tests 5m 🟩
test-target-determinator 5m 🟩
check 4m 🟩
check-dynamic-deps 3m 🟩🟩🟩
rust-move-tests 2m 🟩
rust-move-tests 2m 🟩
rust-move-tests 2m 🟩
fetch-last-released-docker-image-tag 2m 🟩
general-lints 1m 🟩🟩🟩
semgrep/ci 1m 🟩🟩🟩
Backport PR 37s 🟥🟩
file_change_determinator 33s 🟩🟩🟩
file_change_determinator 11s 🟩

settingsfeedbackdocs ⋅ learn more about trunk.io

@vusirikala vusirikala enabled auto-merge (squash) November 1, 2024 23:19

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor

github-actions bot commented Nov 1, 2024

✅ Forge suite realistic_env_max_load success on ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89

two traffics test: inner traffic : committed: 14137.91 txn/s, latency: 2810.22 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 6000 ms), latency samples: 5375580
two traffics test : committed: 100.05 txn/s, latency: 1619.88 ms, (p50: 1400 ms, p70: 1400, p90: 1600 ms, p99: 10900 ms), latency samples: 1840
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.005, avg: 1.603", "ConsensusProposalToOrdered: max: 0.321, avg: 0.297", "ConsensusOrderedToCommit: max: 0.366, avg: 0.358", "ConsensusProposalToCommit: max: 0.668, avg: 0.655"]
Max non-epoch-change gap was: 1 rounds at version 5043411 (avg 0.00) [limit 4], 1.88s no progress at version 5043411 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.52s no progress at version 2648337 (avg 8.52s) [limit 15].
Test Ok

Copy link
Contributor

github-actions bot commented Nov 1, 2024

✅ Forge suite framework_upgrade success on 4ce6dddf9e0cfe007f84cbb4756368295417b3ce ==> ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89

Compatibility test results for 4ce6dddf9e0cfe007f84cbb4756368295417b3ce ==> ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89 (PR)
Upgrade the nodes to version: ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1327.22 txn/s, submitted: 1331.18 txn/s, failed submission: 3.96 txn/s, expired: 3.96 txn/s, latency: 2310.56 ms, (p50: 2100 ms, p70: 2400, p90: 3000 ms, p99: 4000 ms), latency samples: 120520
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1324.26 txn/s, submitted: 1327.17 txn/s, failed submission: 2.91 txn/s, expired: 2.91 txn/s, latency: 2286.64 ms, (p50: 2100 ms, p70: 2400, p90: 3500 ms, p99: 5100 ms), latency samples: 118220
5. check swarm health
Compatibility test for 4ce6dddf9e0cfe007f84cbb4756368295417b3ce ==> ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89 passed
Upgrade the remaining nodes to version: ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1286.47 txn/s, submitted: 1290.45 txn/s, failed submission: 3.98 txn/s, expired: 3.98 txn/s, latency: 2309.58 ms, (p50: 2100 ms, p70: 2600, p90: 3300 ms, p99: 4600 ms), latency samples: 116360
Test Ok

Copy link
Contributor

github-actions bot commented Nov 1, 2024

✅ Forge suite compat success on 4ce6dddf9e0cfe007f84cbb4756368295417b3ce ==> ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89

Compatibility test results for 4ce6dddf9e0cfe007f84cbb4756368295417b3ce ==> ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89 (PR)
1. Check liveness of validators at old version: 4ce6dddf9e0cfe007f84cbb4756368295417b3ce
compatibility::simple-validator-upgrade::liveness-check : committed: 14557.60 txn/s, latency: 2292.35 ms, (p50: 1900 ms, p70: 2100, p90: 2200 ms, p99: 7200 ms), latency samples: 482660
2. Upgrading first Validator to new version: ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6472.72 txn/s, latency: 4281.76 ms, (p50: 4800 ms, p70: 4900, p90: 5500 ms, p99: 5700 ms), latency samples: 130440
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7096.14 txn/s, latency: 4592.96 ms, (p50: 4900 ms, p70: 5000, p90: 6400 ms, p99: 6800 ms), latency samples: 238600
3. Upgrading rest of first batch to new version: ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6685.42 txn/s, latency: 4215.86 ms, (p50: 4900 ms, p70: 5100, p90: 5200 ms, p99: 5200 ms), latency samples: 122720
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6062.24 txn/s, latency: 5142.43 ms, (p50: 5300 ms, p70: 5400, p90: 6500 ms, p99: 7100 ms), latency samples: 226840
4. upgrading second batch to new version: ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 8926.33 txn/s, latency: 3153.54 ms, (p50: 3500 ms, p70: 3700, p90: 4200 ms, p99: 4500 ms), latency samples: 155500
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8453.58 txn/s, latency: 3759.70 ms, (p50: 3800 ms, p70: 4000, p90: 5700 ms, p99: 6400 ms), latency samples: 274860
5. check swarm health
Compatibility test for 4ce6dddf9e0cfe007f84cbb4756368295417b3ce ==> ac57ca1c2070b22a8d5b4f48dbc2dfc7ae3b6f89 passed
Test Ok

@vusirikala vusirikala merged commit 3cad55d into main Nov 1, 2024
142 of 143 checks passed
@vusirikala vusirikala deleted the balaji/timeout-author branch November 1, 2024 23:51
github-actions bot pushed a commit that referenced this pull request Nov 1, 2024
Copy link
Contributor

github-actions bot commented Nov 1, 2024

💚 All backports created successfully

Status Branch Result
aptos-release-v1.23

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

vusirikala pushed a commit that referenced this pull request Nov 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants