[Monit] Unmonitor the processes in containers which are disabled.

[yozhao101]

- Why I did it
We want to let Monit to unmonitor the processes in containers which are disabled in FEATURE table such that
Monit will not generate false alerting messages into the syslog.

- How I did it
Monit will periodically run a script which accepts three parameters: <container_name>, <process_name> and
<process_cmdline>. This script will first check whether the container is disabled in the FEATURE table or not.
If it is disabled, Monit will skip monitoring the processes. Otherwise, this script will leverage psutil library to inspect
the process tree in host to look for the processes. If the process is not found, then an alerting message will be written
into syslog.

- How to verify it
We can change the state field of a container in FEATURE table from enabled to disabled and then kill a critical
process in it to see whether Monit can generate the alerting message in syslog or not. The message format in syslog is:
<process_name> is not running.

- Which release branch to backport (provide reason below if selected)

• 201811
• 201911
• 202006

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

[abdosi]

@yozhao101 @jleveque

While looking into fix for Issue##5292 I came across this PR. I am wondering it will be easier we can use 'group' concept of monit. We can control monitor/unmonitor of this group from hostcfgd same as how we start/unmask and stop/mask the service. I verified locally and looks good.

Updated Telemtry monit file with group

check process telemetry matching "/usr/sbin/telemetry"
if does not exist for 5 times within 5 cycles then alert
group telemetry.service

check process dialout_client matching "/usr/sbin/dialout_client_cli"
if does not exist for 5 times within 5 cycles then alert
group telemetry.service

Now when Feature is disabled we can execute sudo monit -g telemetry.service unmonitor from hostcfgd.
And when Feature is enabled we can execute sudo monit -g telemetry.service monitor from hostcfgd.

Any concern in this approach ?

[jleveque]

@abdosi: Thank you for this suggestion. I think it is a feasable solution. The only concern I have is we would then spread the need for knowledge of Monit to other components (in this case, hostcfgd). I would prefer to encapsulate and abstract this knowledge as much as possible, but I wouldn't say it's a dealbreaker.

I think we can go ahead with this PR for the time being, and @yozhao101 can investigate the 'group' approach that you propose as a future cleanup/enhancement.

[abdosi]

@abdosi: Thank you for this suggestion. I think it is a feasable solution. The only concern I have is we would then spread the need for knowledge of Monit to other components (in this case, hostcfgd). I would prefer to encapsulate and abstract this knowledge as much as possible, but I wouldn't say it's a dealbreaker.

I think we can go ahead with this PR for the time being, and @yozhao101 can investigate the 'group' approach that you propose as a future cleanup/enhancement.

@jleveque Thanks. I felt since hostcfgd already has awareness with Feature Table and Services Enable/Disable monit going there makes it more logical.

[yozhao101]

@abdosi: Thank you for this suggestion. I think it is a feasable solution. The only concern I have is we would then spread the need for knowledge of Monit to other components (in this case, hostcfgd). I would prefer to encapsulate and abstract this knowledge as much as possible, but I wouldn't say it's a dealbreaker.

I think we can go ahead with this PR for the time being, and @yozhao101 can investigate the 'group' approach that you propose as a future cleanup/enhancement.

@jleveque Thanks so much for helping me answering this question. I will investigate the group approach and give the feedback.

[jleveque]

Check builds are failing and there are conflicts. Please address.

[yozhao101]

Check builds are failing and there are conflicts. Please address.

Fixed.

[jleveque]

Retest baseimage please

[jleveque]

Retest vsimage please

[jleveque]

Retest vsimage please

[jleveque]

@yozhao101: Will this PR cherry-pick cleanly to 201911, or will you need to open a separate PR?

[yozhao101]

@yozhao101: Will this PR cherry-pick cleanly to 201911, or will you need to open a separate PR?

I will try cherry-pick this PR to 201911.

[yozhao101]

Retest vsimage please

[yozhao101]

Retest vsimage please

[yozhao101]

Retest vsimage please