[config] Eliminate race condition between reloading Monit config and monitoring container_checker #1543
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
enabling container_checker. Signed-off-by: Yong Zhao <[email protected]>
yozhao101
changed the title
|
@yozhao101: Please add the "Request for " label if requesting a cherry-pick. the "Included in label is used to indicate cherry-pick has been done, or if a PR is targeted at a release branch. |
jleveque
approved these changes
Apr 2, 2021
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Thank Joe! I will pay attention to such issue in future. |
yxieca
pushed a commit
that referenced
this pull request
Apr 8, 2021
…1543) Signed-off-by: Yong Zhao [email protected] What I did Nightly test found a failure when we ran the command sudo config reload/load_minigraph, The error message is: admin@str-a7050-acs-1:~$ sudo config reload -y Disabling container monitoring ... Stopping SONiC target ... Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db Running command: /usr/local/bin/db_migrator.py -o migrate Resetting failed status on bgp.service Resetting failed status on caclmgrd.service Resetting failed status on dhcp_relay.service Resetting failed status on hostcfgd.service Resetting failed status on hostname-config.service Resetting failed status on interfaces-config.service Resetting failed status on lldp.service Resetting failed status on ntp-config.service Resetting failed status on pmon.service Resetting failed status on procdockerstatsd.service Resetting failed status on radv.service Resetting failed status on rsyslog-config.service Resetting failed status on swss.service Resetting failed status on syncd.service Resetting failed status on teamd.service Resetting failed status on telemetry.timer Restarting SONiC target ... Reloading Monit configuration ... Reinitializing monit daemon Enabling container monitoring ... Unix socket /var/run/monit.sock connection error -- No such file or directory The root reason is that there exists an implicit race condition between the command sudo monit reload at line 701 and the command sudo monit monitor container_checker at line 706. Both commands need access the monit.sock socket file under the directory /var/run/. Specifically the sudo monit reload at line 701 will re-initialize the Monit daemon, delete old monit.sock file and then create a new one. During this re-initializing process, the command sudo monit status can always execute successfully at line 704 before the old monit.sock file was deleted, but the command sudo monit monitor container_checker at line 706 will only succeed if the new monit.sock was created, otherwise it will fail and raise this error message. How I did it I changed the sequence between the operation to reload Monit configuration and the operation to enable monitoring container_checker. How to verify it I verified this change on DuT str-a7050-acs-1 by running the command sudo config reload/load_minigraph -y to make sure the error was not raised again. Previous command output (if the output of a command-line utility has changed) admin@str-a7050-acs-1:~$ sudo config reload -y Disabling container monitoring ... Stopping SONiC target ... Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db Running command: /usr/local/bin/db_migrator.py -o migrate Resetting failed status on bgp.service Resetting failed status on caclmgrd.service Resetting failed status on dhcp_relay.service Resetting failed status on hostcfgd.service Resetting failed status on hostname-config.service Resetting failed status on interfaces-config.service Resetting failed status on lldp.service Resetting failed status on ntp-config.service Resetting failed status on pmon.service Resetting failed status on procdockerstatsd.service Resetting failed status on radv.service Resetting failed status on rsyslog-config.service Resetting failed status on swss.service Resetting failed status on syncd.service Resetting failed status on teamd.service Resetting failed status on telemetry.timer Restarting SONiC target ... Reloading Monit configuration ... Reinitializing monit daemon Enabling container monitoring ... New command output (if the output of a command-line utility has changed) admin@str-a7050-acs-1:~$ sudo config reload -y Disabling container monitoring ... Stopping SONiC target ... Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db Running command: /usr/local/bin/db_migrator.py -o migrate Resetting failed status on bgp.service Resetting failed status on caclmgrd.service Resetting failed status on dhcp_relay.service Resetting failed status on hostcfgd.service Resetting failed status on hostname-config.service Resetting failed status on interfaces-config.service Resetting failed status on lldp.service Resetting failed status on ntp-config.service Resetting failed status on pmon.service Resetting failed status on procdockerstatsd.service Resetting failed status on radv.service Resetting failed status on rsyslog-config.service Resetting failed status on swss.service Resetting failed status on syncd.service Resetting failed status on teamd.service Resetting failed status on telemetry.timer Restarting SONiC target ... Enabling container monitoring ... Reloading Monit configuration ... Reinitializing monit daemon
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Yong Zhao [email protected]
What I did
Nightly test found a failure when we ran the command
sudo config reload/load_minigraph, The error message is:The root reason is that there exists an
implicit race conditionbetween the commandsudo monit reloadat line 701 andthe command
sudo monit monitor container_checkerat line 706. Both commands need access themonit.socksocket fileunder the directory
/var/run/.Specifically the
sudo monit reloadat line 701 will re-initialize the Monit daemon, delete oldmonit.sockfile and then create a new one. During this re-initializing process, the commandsudo monit statuscan always execute successfully at line 704 before the oldmonit.sockfile was deleted, but the commandsudo monit monitor container_checkerat line 706 will only succeed if the newmonit.sockwas created, otherwise it will fail and raise this error message.How I did it
I changed the sequence between the operation to reload Monit configuration and the operation to enable monitoring container_checker.
How to verify it
I verified this change on DuT
str-a7050-acs-1by running the commandsudo config reload/load_minigraph -yto make sure the error was not raised again.Previous command output (if the output of a command-line utility has changed)
New command output (if the output of a command-line utility has changed)