[system-health] Make check interval more accurate #14085
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
keboliu
approved these changes
Mar 6, 2023
|
@prgeor could you please help to review or assign someone to do so? |
|
@Junchao-Mellanox can you elaborate more on this |
Currently, sonic-mgmt has a test case like this:
The case failed intermittently because sometimes 70 seconds is not enough. If we introduce a time compensation to healthd main loop, healthd loop interval will be more accurate and 70 seoncds is enough. BTW, time compensation is a common way in sonic. |
prgeor
reviewed
Mar 14, 2023
|
|
||
| if self.stop_event.wait(manager.config.interval): | ||
| break | ||
| while self._run_checker(manager, chassis): |
There was a problem hiding this comment.
previously healthd was running continuously in while 1 loop. with this change the loop can break and healthd wont run. why this is changed?
There was a problem hiding this comment.
Previous the loop still can be break, this logic does not change actually.
while 1:
stat = manager.check(chassis)
self._process_stat(chassis, manager.config, stat)
if self.stop_event.wait(manager.config.interval):
break
| self.log_notice(f'System health takes {elapse} seconds for one iteration') | ||
| sleep_time_in_sec = 1 | ||
| if self.stop_event.wait(sleep_time_in_sec): | ||
| return False |
There was a problem hiding this comment.
if return 'False' healthd will stop. can we just break?
There was a problem hiding this comment.
if stop_event is set, it means that healthd got a signal, and it should stop.
prgeor
approved these changes
Mar 15, 2023
|
@liat-grozovik @lguohan please help merge |
liat-grozovik
approved these changes
Mar 15, 2023
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Mar 15, 2023
- Why I did it Healthd check system status every 60 seconds. However, running checker may take several seconds. Say checker takes X seconds, healthd takes (60 + X) seconds to finish one iteration. This implementation makes sonic-mgmt test case not so stable because the value X is hard to predict and different among different platforms. This PR introduces an interval compensation mechanism to healthd main loop. - How I did it Introduces an interval compensation mechanism to healthd main loop: healthd should wait (60 - X) seconds for next iteration - How to verify it Manual test Unit test
|
Cherry-pick PR to 202205: #14256 |
mssonicbld
pushed a commit
that referenced
this pull request
Mar 15, 2023
- Why I did it Healthd check system status every 60 seconds. However, running checker may take several seconds. Say checker takes X seconds, healthd takes (60 + X) seconds to finish one iteration. This implementation makes sonic-mgmt test case not so stable because the value X is hard to predict and different among different platforms. This PR introduces an interval compensation mechanism to healthd main loop. - How I did it Introduces an interval compensation mechanism to healthd main loop: healthd should wait (60 - X) seconds for next iteration - How to verify it Manual test Unit test
mssonicbld
added
Included in 202205 Branch
and removed
Approved for 202205 Branch
Created PR to 202205 Branch
labels
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Mar 19, 2023
- Why I did it Healthd check system status every 60 seconds. However, running checker may take several seconds. Say checker takes X seconds, healthd takes (60 + X) seconds to finish one iteration. This implementation makes sonic-mgmt test case not so stable because the value X is hard to predict and different among different platforms. This PR introduces an interval compensation mechanism to healthd main loop. - How I did it Introduces an interval compensation mechanism to healthd main loop: healthd should wait (60 - X) seconds for next iteration - How to verify it Manual test Unit test
|
Cherry-pick PR to 202211: #14326 |
mssonicbld
pushed a commit
that referenced
this pull request
Mar 19, 2023
- Why I did it Healthd check system status every 60 seconds. However, running checker may take several seconds. Say checker takes X seconds, healthd takes (60 + X) seconds to finish one iteration. This implementation makes sonic-mgmt test case not so stable because the value X is hard to predict and different among different platforms. This PR introduces an interval compensation mechanism to healthd main loop. - How I did it Introduces an interval compensation mechanism to healthd main loop: healthd should wait (60 - X) seconds for next iteration - How to verify it Manual test Unit test
mssonicbld
added
Included in 202211 Branch
and removed
Approved for 202211 Branch
Created PR to 202211 Branch
labels
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I did it
Healthd check system status every 60 seconds. However, running checker may take several seconds. Say checker takes X seconds, healthd takes (60 + X) seconds to finish one iteration. This implementation makes sonic-mgmt test case not so stable because the value X is hard to predict and different among different platforms. This PR introduces an interval
compensation mechanism to healthd main loop.
How I did it
Introduces an interval compensation mechanism to healthd main loop: healthd should wait (60 - X) seconds for next iteration
How to verify it
Manual test
Unit test
Which release branch to backport (provide reason below if selected)
Description for the changelog
Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)