[Feature Request] Output VM kernel logs to host

[awprice]

Is your feature request related to a problem? Please describe.

We would like to have access to the logs from the Kernel running inside the Kata VMs available on the host for debugging purposes. We currently use the console to access the VM to view the logs, but ideally in the future we would like to disable console access (for security reasons) and have the logs accessible on the host.

Describe the solution you'd like

Kernel logs (the sort you get from dmesg) saved onto the node in a central location, e.g. /var/logs/ etc.

Describe alternatives you've considered

We could run a sidecar (an additional container) in our pod that tails dmesg, but we would prefer a more elegant solution that doesn't require running an extra container for exporting kernel logs.

Additional context

N/A

Before raising this feature request

Have you looked at the limitations document? Yes

[jodh-intel]

The guest kernels logs are already available on the host. Assuming you are using the default configuration (using a kata-proxy), the proxy logs its messages to the hosts journal. And the proxy includes messages from:

• itself
• the guest kernel
• the agent

Since all of the above message categories get logged to the journal by process kata-proxy, it is necessary to "unpack" the guest kernel logs from inside the kata-proxy log messages. See: https://github.com/kata-containers/tests/tree/master/cmd/log-parser

[awprice]

@jodh-intel We are using the default configuration, and from what I understand we are using shimv2. Based on this diagram, it looks like kata-proxy isn't used with shimv2?

(From https://github.com/kata-containers/documentation/blob/master/design/architecture.md#overview)

When starting my Kata containers through Kubernetes, I also don't see a kata-proxy process running on the host node nor can find any VM kernel logs in journal.

[lifupan]

Hi @awprice
For shimv2, if you didn't use vsock, it would use a builtIn proxy, which means there is no separated proxy process running. If you want to get the dmesg log, you can:

1. set kernel_params = "agent.log=debug" in kata configure file,
2. set enable_debug = true in [runtime] section in kata configure file.

Then you will get the proxy/dmesg logs in containerd's log.

[awprice]

@lifupan I tried your instructions above, and unfortunately couldn't get it to work. I started seeing the proxy logs, but the dmesg logs weren't in containerd's logs. Is there any special VM/kernel configuration needed?

It also seemed to stop console access to the VM working when enabling enable_debug = true in the [runtime] section.

[lifupan]

@lifupan I tried your instructions above, and unfortunately couldn't get it to work. I started seeing the proxy logs, but the dmesg logs weren't in containerd's logs. Is there any special VM/kernel configuration needed?

Maybe I misunderstand you, by this method, the proxy would print the qemu's console into logs, including the guest boot logs, isn't that what you want?

It also seemed to stop console access to the VM working when enabling enable_debug = true in the [runtime] section.

Yes, since you enabled debug, the proxy will open the console and direct the console's output into containerd's log.

[awprice]

Maybe I misunderstand you, by this method, the proxy would print the qemu's console into logs, including the guest boot logs, isn't that what you want?

Yep that is what I want, but it doesn't seem to occur. Is there any other configuration needed?

[lifupan]

@lifupan I tried your instructions above, and unfortunately couldn't get it to work. I started seeing the proxy logs,

What's the proxy logs did you see? is there any log lines prefixed with "vmconsole="?

[jodh-intel]

The debug console / proxy log issue is documented in the Developer guide, so I'd really suggest reading that carefully:

• https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#ensure-debug-options-are-valid

[awprice]

@lifupan Apologies, I've now located the vmconsole= lines, and found the kernel logs. Thanks for the help @lifupan, @jodh-intel.

[egernst]

@jodh-intel in a non proxy world, this should be feasible through traces, right?

[jodh-intel]

? Converting guest kernel messages to traces?

[lifupan]

I think we can even catch the vm console output in the noproxy, is it better to add a catch function to noproxy?