Send GitLab host logs to CloudWatch

[amarjandu]

/mnt/gitlab/logs/
/var/log/

[achave11-ucsc]

@hannes-ucsc:

Perform following spike:

1. Build docker image for agent https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-build-docker-image.html
2. Start EC2 instance from rancher AMI that's used by our GitLab instance (@amarjandu can help).
3. Upload image tarball to EC2 instance
4. Import tarball
5. Configure agent
6. Start container from image
7. Verify that logs are send to cloudwatch

It's importat to be aware of the distinction between the deprecated and the new agent. See note at top of page https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html.

[hannes-ucsc]

Change of plans: we'll be migrating the GitLab instance to start from the Amazon Linux AMI on which the CloudWatch agent can be installed via yum.

[hannes-ucsc]

Docker uses the syslog driver by default. On the Amazon Linux AMI, this causes all container output (stderr and stdout) to be written to /var/log/messages. The container running the gitlab image on that instance runs a wrapper which runs gitlab-ctl tail in the background, which runs Unix tail on all log files written to /var/log/gitlab in the container (/mnt/gitlab/logs on the host). The tail command is invoked with --name which causes every line in tail's output to be prefaced with a line reading ===> /var/log/gitlab/foo.log. This is relatively useless because the adjacency correlation gets lost in /var/log/messages on the host with interspersed lines from other processes.

We should disable the tail by setting GITLAB_SKIP_TAIL_LOGS for the gitlab container so that the verbose GitLab container logs don't pollute /var/log/messages. They are already available on the host under /mnt/gitlab/logs from where the CloudWatch Logs agent can be configured pick them up.

Scan all file system for other log directories we might want to forward to CloudWatch. The SSM agent log directory comes to mind, for example.

[achave11-ucsc]

All the log entries agreed on were added, and are active. Additionally, the experiment on …/reconfigure/{dated}.log was successful, not only is the log forwarded after starting and stopping the gitlab deamon (and dependents), but the log forwards the re-creation of the instance (when new log entries and related changes were pushed) as well.

[hannes-ucsc]

For demo, show that the forwarding of the logs survives restarting the agent (via systemctl restart) as well as rebooting and recreating the instance. Show evidence the GITLAB_SKIP_TAIL_LOGS does indeed prevent GitLab log files from being tailed or sent to the host. Show that the agent picks up new log files matching /mnt/gitlab/logs/reconfigure/*.log without having to be restarted. I think restarting the gitlab container or running gitlab-ctl reconfigure inside it causes a new log file to be created.

https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure

[nolunwa-ucsc]

@dsotirho-ucsc Please update this spreadsheet to include the Gitlogs audit event captured in cloudwatch. https://docs.google.com/spreadsheets/d/1XTl3tQmr9wU1o8z8uGNjOU55jYlmWWMBR33QuKfg_S4/edit#gid=311132757