fix(ci): Avoid inbound service overloads and fix failing tests #6537
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
teor2345
added
P-High 🔥
I-hang
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #6537 +/- ##
==========================================
- Coverage 77.92% 77.74% -0.19%
==========================================
Files 306 306
Lines 40236 40267 +31
==========================================
- Hits 31355 31305 -50
- Misses 8881 8962 +81 |
This disables around 10,000 logs like:
2023-04-18T02:46:28.441662Z WARN init{config=Config { checkpoint_sync: true, debug_skip_parameter_preload: false } network=Mainnet debug_skip_parameter_preload=true}: unexpected error: Closed in state request while verifying previous state checkpoints
teor2345
force-pushed
the
inbound-overload-fix
branch
from
0e8cff7 to
8ed898b
Compare
oxarbitrage
approved these changes
Apr 23, 2023
teor2345
changed the title
teor2345
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
It looks like the syncer or state is hanging, then the inbound service is getting overloaded, which causes CI failures in some tests.
This might fix #6506.
Complex Code or Requirements
It's probably a concurrency bug, but this PR just tweaks some timings and test logs.
Solution
Fixes:
Diagnostics:
Review
This is an urgent fix for failures that are stopping other PRs merging.
Reviewer Checklist
Follow Up Work
We might need to fix a state or syncer hang, but we need more debug info to know which one to fix.