[S6100] Improve S6100 serial-getty monitor, wait and re-check when ge…

…tty not running to avoid false alert. (#14402) (#14755)

[S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert. 

This is cherry-pick PR for: #14402

#### Why I did it
On S6100, the serial-getty service some time can't auto-restart by systemd. So there is a monit unit to check serial-getty service status and restart it.

However, this monit will report false alert, because in most case when serial-getty not running, systemd can restart it successfully.

To avoid the false alert, improve the monitor to wait and re-check.

Steps to reproduce this issue:
1. User login to device via console, and keep the connection.
2. User login to device via SSH, check the [email protected] service, it's running.
3. Run 'monit reload' from SSH connection.
4. Check syslog 1 minutes later, there will be false alert: ' 'serial-getty' process is not running'

##### Work item tracking
- Microsoft ADO :17424426

#### How I did it
Add check-getty.sh script to recheck again later when getty service not running.
And update monit unit to check serial-getty service status with this script to avoid false alert.

#### How to verify it
Pass all UT.
Manually check fixed code work correctly:


```
admin@***:~$ sudo systemctl stop  [email protected]
admin@***:~$ sudo /usr/local/bin/check-getty.sh 
admin@***:~$ echo $?
1
admin@***:~$ sudo systemctl status [email protected]
● [email protected] - Serial Getty on ttyS1
     Loaded: loaded (/lib/systemd/system/[email protected]; enabled-runtime; vendor preset: enabled)
     Active: inactive (dead) since Tue 2023-03-28 07:15:21 UTC; 1min 13s ago

admin@***:~$ sudo /usr/local/bin/check-getty.sh 
admin@***:~$ echo $?
0
admin@***:~$ sudo systemctl status [email protected]
● [email protected] - Serial Getty on ttyS1
     Loaded: loaded (/lib/systemd/system/[email protected]; enabled-runtime; vendor preset: enabled)
```

syslog:
```
Mar 28 07:10:37.597458 *** INFO systemd[1]: [email protected]: Succeeded.
Mar 28 07:12:43.010550 *** ERR monit[593]: 'serial-getty' status failed (1) -- no output
Mar 28 07:12:43.010744 *** INFO monit[593]: 'serial-getty' trying to restart
Mar 28 07:12:43.010846 *** INFO monit[593]: 'serial-getty' stop: '/bin/systemctl stop [email protected]'
Mar 28 07:12:43.132172 *** INFO monit[593]: 'serial-getty' start: '/bin/systemctl start [email protected]'
Mar 28 07:13:43.286276 *** INFO monit[593]: 'serial-getty' status succeeded (0) -- no output
```

#### Tested branch (Please provide the tested image version)

- [x] 20201231.77

#### Description for the changelog
[S6100] Improve S6100 serial-getty monitor.

platform/broadcom/sonic-platform-modules-dell/debian/platform-modules-s6100.install

@@ -14,6 +14,7 @@ s6100/scripts/soft-reboot_plugin usr/share/sonic/device/x86_64-dell_s6100_c2538-
 s6100/scripts/ssd-fw-upgrade usr/share/sonic/device/x86_64-dell_s6100_c2538-r0
 s6100/scripts/override.conf /etc/systemd/system/systemd-reboot.service.d
 s6100/scripts/s6100_serial_getty_monitor etc/monit/conf.d
+s6100/scripts/check-getty.sh usr/local/bin
 common/dell_lpc_mon.sh usr/local/bin
 s6100/scripts/s6100_ssd_mon.sh usr/local/bin
 s6100/scripts/s6100_ssd_upgrade_status.sh usr/local/bin

platform/broadcom/sonic-platform-modules-dell/s6100/scripts/check-getty.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+RETRY=0
+while [ $RETRY -lt 5 ]; do
+    let RETRY=$RETRY+1
+
+    /bin/systemctl --quiet is-active [email protected]
+    status=$?
+    if [ $status == 0 ]; then
+        exit 0
+    fi
+
+    # when serial-getty not running, recheck later, beause systemd will restart serial-getty automatically.
+    sleep 1
+done
+
+exit 1

platform/broadcom/sonic-platform-modules-dell/s6100/scripts/s6100_serial_getty_monitor

@@ -1,4 +1,5 @@
 #Dell S6100 serial getty monitor
-check process serial-getty matching "ttyS"
+check program serial-getty with path /usr/local/bin/check-getty.sh
 start program = "/bin/systemctl start [email protected]"
 stop program = "/bin/systemctl stop [email protected]"
+if status != 0 then restart