[202012] [S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert. #14755
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tty not running to avoid false alert. (sonic-net#14402) [S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert. On S6100, the serial-getty service some time can't auto-restart by systemd. So there is a monit unit to check serial-getty service status and restart it. However, this monit will report false alert, because in most case when serial-getty not running, systemd can restart it successfully. To avoid the false alert, improve the monitor to wait and re-check. Steps to reproduce this issue: 1. User login to device via console, and keep the connection. 2. User login to device via SSH, check the [email protected] service, it's running. 3. Run 'monit reload' from SSH connection. 4. Check syslog 1 minutes later, there will be false alert: ' 'serial-getty' process is not running' Add check-getty.sh script to recheck again later when getty service not running. And update monit unit to check serial-getty service status with this script to avoid false alert. Pass all UT. Manually check fixed code work correctly: ``` admin@***:~$ sudo systemctl stop [email protected] admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 1 admin@***:~$ sudo systemctl status [email protected] ● [email protected] - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/[email protected]; enabled-runtime; vendor preset: enabled) Active: inactive (dead) since Tue 2023-03-28 07:15:21 UTC; 1min 13s ago admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 0 admin@***:~$ sudo systemctl status [email protected] ● [email protected] - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/[email protected]; enabled-runtime; vendor preset: enabled) ``` syslog: ``` Mar 28 07:10:37.597458 *** INFO systemd[1]: [email protected]: Succeeded. Mar 28 07:12:43.010550 *** ERR monit[593]: 'serial-getty' status failed (1) -- no output Mar 28 07:12:43.010744 *** INFO monit[593]: 'serial-getty' trying to restart Mar 28 07:12:43.010846 *** INFO monit[593]: 'serial-getty' stop: '/bin/systemctl stop [email protected]' Mar 28 07:12:43.132172 *** INFO monit[593]: 'serial-getty' start: '/bin/systemctl start [email protected]' Mar 28 07:13:43.286276 *** INFO monit[593]: 'serial-getty' status succeeded (0) -- no output ``` [S6100] Improve S6100 serial-getty monitor.
12 tasks
qiluo-msft
approved these changes
Apr 21, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert.
This is cherry-pick PR for: #14402
Why I did it
On S6100, the serial-getty service some time can't auto-restart by systemd. So there is a monit unit to check serial-getty service status and restart it.
However, this monit will report false alert, because in most case when serial-getty not running, systemd can restart it successfully.
To avoid the false alert, improve the monitor to wait and re-check.
Steps to reproduce this issue:
Work item tracking
How I did it
Add check-getty.sh script to recheck again later when getty service not running.
And update monit unit to check serial-getty service status with this script to avoid false alert.
How to verify it
Pass all UT.
Manually check fixed code work correctly:
syslog:
Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
Description for the changelog
[S6100] Improve S6100 serial-getty monitor.
Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)