CRI Logging Driver

[nwneisen]

Fixes #246, #213
Replaces #224

Proposed Changes

This PR introduces a CRI logging driver as an alternative solution to #213. This driver outputs docker logs in CRI formats. Docker has dual logging capability that will take care of also outputting the logs in JSON format.

The driver creates a CRI logger to output CRI formatted logs to the location specified by kubelet. This logger will exist within cri-dockerd. i.e. cri-dockerd will handle it's current GRPC messages as well as the logger messages from dockerd

The driver will need to be installed with cri-dockerd and specified in cri-dockerd's code when it creates a container along with the kube file location. The kube file location is passed to the driver to eliminated the need to create a kube client within the driver when it already exists in cri-dockerd.

Changes Still Needed

• Build and install the driver with cri-dockerd
  The driver will now run as a part of cri-dockerd. My thoughts are to have cri-dockerd tell dockerd it is installed
  using something like https://docs.docker.com/engine/api/v1.43/#tag/Plugin/operation/PluginPull.
• Change cri-dockerd to specify the driver and kube path when creating a container
• Create documentation

[zmedico]

Docker has dual logging capability that will take care of also outputting the logs in JSON format.

I've tried dual logging before, using the fluentd driver, and for dual logging docker used the "local" driver as a "dual logging cache" instead of the json-file driver. This broke kubectl logs, since it was looking for the json-file logs which the "local" driver does not produce. Meanwhile, docker logs was still functional, since it was apparently able to work with the "local" driver.

[nwneisen]

@zmedico The current logging code for cri-dockerd creates a symlink to the json log file created by docker in the log file location that is expected/designated by kubectl. This code would have to stay in place for the dual logging to work. My guess is that the fluentd driver does not create this symlink and therefor kubectl cannot find the log file when it tries to read the logs.

[zmedico]

@zmedico The current logging code for cri-dockerd creates a symlink to the json log file created by docker in the log file location that is expected/designated by kubectl.

Yes, I've found the createContainerLogSymlink function in cri-dockerd/core/logs.go, and from the cri-dockerd debug log it seems that the path variable refers to a json-file style path that does not exist when the fluentd log driver is used. Ultimately it logs an Unsupported logging driver by CRI message since the IsCRISupportedLogDriver function returns false for any driver except json-file. The only log file that actually exists for the container is named container-cached.log and is a different format that docker's local logging driver uses for dual-logging (decoder found in moby/daemon/logger/local/read.go).

This code would have to stay in place for the dual logging to work. My guess is that the fluentd driver does not create this symlink and therefor kubectl cannot find the log file when it tries to read the logs.

My fear is that using your criLogDriver will give similar results, since docker will still use its local logging driver for dual-logging, and the createContainerLogSymlink will behave approximately the same as it does with the fluentd driver.

The "fallback" method that you mention in #246 (comment) would avoid this problem, since docker would continue to use the json-file log driver which is apparently the only log driver compatible with the createContainerLogSymlink function.

I suppose if criLogDriver writes its log to the same realPath that the json-file driver would have used, then that might just work, since kubectl logs can read cri format.