[6.1.x] Collect gravity cli history

[bernardjkim]

Description

• Gravity cli commands will be recorded in the system journal and tagged with gravity-cli. Logs can be queried for with journalctl -t gravity-cli.
• Gravity cli history will also be included in the gravity report.
• Potentially sensitve data will be redacted from the logs and values replaced with *****.
• The start/ffinish of a particular command can be tracked using the proc id, journalctl _PID=[pid].

Type of change

• New feature (non-breaking change which adds functionality)

Linked tickets and other PRs

• Ports Collect gravity cli history #1845, Sanitize gravity cli cmd logs #1889
• Requires https://github.com/gravitational/gravity.e/pull/4310

TODOs

• Self-review the change
• Perform manual testing
• Address review feedback

Testing done

Verify gravity cli is recorded in system journal

Verfiy gravity cli history is included in gravity report

[vagrant@node-1 ~]$ cat gravity-cli.log
-- Logs begin at Fri 2020-07-10 20:26:59 UTC, end at Mon 2020-07-13 04:28:00 UTC. --
Jul 13 04:27:55 node-1 gravity-cli[5646]: [RUNNING]: /usr/bin/gravity system report --insecure --filter "etcd" --compressed
Jul 13 04:27:55 node-1 gravity-cli[5661]: [RUNNING]: /usr/bin/gravity planet enter -- --notty /usr/bin/curl -- -s --tlsv1.2 --cacert /var/state/root.cert --cert /var/state/etcd.cert --key /var/state/etcd.key https:/127.0.0.1:2379/metrics
Jul 13 04:27:56 node-1 gravity-cli[5646]: [SUCCESS]: /usr/bin/gravity system report --insecure --filter "etcd" --compressed
Jul 13 04:27:56 node-1 gravity-cli[5698]: [RUNNING]: /usr/bin/gravity system report --insecure --filter "system" --compressed --since "10s"
Jul 13 04:27:56 node-1 gravity-cli[5729]: [RUNNING]: /usr/bin/gravity planet enter -- --notty /sbin/bridge -- fdb show
Jul 13 04:27:58 node-1 gravity-cli[5799]: [RUNNING]: /usr/bin/gravity planet enter -- --notty /usr/bin/etcdctl -- cluster-health
Jul 13 04:27:58 node-1 gravity-cli[5843]: [RUNNING]: /usr/bin/gravity planet enter -- --notty /usr/bin/etcdctl3 -- endpoint health --cluster
Jul 13 04:27:58 node-1 gravity-cli[5898]: [RUNNING]: /usr/bin/gravity planet enter -- --notty /usr/bin/planet -- status
Jul 13 04:27:59 node-1 gravity-cli[5943]: [RUNNING]: /usr/bin/gravity planet enter -- --notty /bin/systemctl -- status --full
Jul 13 04:27:59 node-1 gravity-cli[5977]: [RUNNING]: /usr/bin/gravity planet enter -- --notty /bin/systemctl -- --failed --full
Jul 13 04:27:59 node-1 gravity-cli[6012]: [RUNNING]: /usr/bin/gravity planet enter -- --notty /bin/systemctl -- list-jobs --full
Jul 13 04:27:59 node-1 gravity-cli[6047]: [RUNNING]: /usr/bin/gravity planet enter -- --notty /bin/systemctl -- list-jobs --full
Jul 13 04:27:59 node-1 gravity-cli[6084]: [RUNNING]: /usr/bin/gravity planet enter -- --notty /usr/bin/serf -- members
Jul 13 04:28:00 node-1 gravity-cli[6136]: [RUNNING]: /usr/bin/gravity system export-runtime-journal --since "10s"
Jul 13 04:28:00 node-1 gravity-cli[6148]: [RUNNING]: /usr/bin/gravity system stream-runtime-journal --since "10s"
Jul 13 04:28:00 node-1 gravity-cli[6136]: [SUCCESS]: /usr/bin/gravity system export-runtime-journal --since "10s"

Verify log entry of a successfully completed gravity command

Jul 13 04:24:48 node-1 gravity-cli[4607]: [RUNNING]: /usr/bin/gravity status cluster
Jul 13 04:24:48 node-1 gravity-cli[4607]: [SUCCESS]: /usr/bin/gravity status cluster

Verify log entry of a failed gravity command. Also verify sensitive data is redacted

Jul 13 04:24:56 node-1 gravity-cli[4642]: [RUNNING]: /usr/bin/gravity install --token "*****"
Jul 13 04:24:56 node-1 gravity-cli[4642]: [FAILURE]: /usr/bin/gravity install --token "*****": [ERROR]: install token is too short, min length is 6