Use RWMutex in demultiplexer to prevent check delays #26148
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gh123man
added
changelog/no-changelog
team/agent-metrics-logs
qa/done
gh123man
changed the title
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #26148 +/- ##
===========================================
+ Coverage 45.06% 57.70% +12.63%
===========================================
Files 2338 778 -1560
Lines 269660 64798 -204862
===========================================
- Hits 121532 37393 -84139
+ Misses 138511 25505 -113006
+ Partials 9617 1900 -7717
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Test changes on VMUse this command from test-infra-definitions to manually test this PR changes on a VM: inv create-vm --pipeline-id=35487830 --os-family=ubuntu |
vickenty
approved these changes
May 30, 2024
|
/merge |
|
🚂 MergeQueue Pull request added to the queue. There are 2 builds ahead! (estimated merge in less than 1h) Use |
dd-mergequeue
bot
deleted the
brian/demultiplexer-agent-sender-rwlock-main
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Followup to this PR: #25858
Very long running flushes can cause the lock in
demultiplexer_agentto be held while checks are starting. When checks start they callGetSenderto acquire their sender instance which shares the same lock as flush. As a result, a very long running flush will block all checks from starting until it completes.In order to avoid check execution delays, we can use an RWMutex here since the protected state is only mutated during
Stop. SoRLockwill never block during normal agent operations allowing checks to start asynchronously of the flush operation.Motivation
Additional Notes
Possible Drawbacks / Trade-offs
Describe how to test/QA your changes
This was deployed in a large staging cluster so QA is done.