Add the ability to read aa_kbc_params from agent-config file

[stevenhorsman]

In peer pods (and it is possible in local hypervisor config too), we use an agent-config.toml as a way to get the agent configuration from a file on the 'guest image' without going through kernel_params. It is currently the only way to restrict endpoints and used to be the only way to pass aa_kbc_params, so we use it quite a bit on our test files. https://github.com/kata-containers/kata-containers/blob/CCv0/tools/osbuilder/rootfs-builder/agent-config.toml.in is an example of the format and https://github.com/kata-containers/kata-containers/blob/ee15a389de021acf2de96e372761dd6271212c16/src/agent/src/config.rs#L420-L429 is where we process it. By convention the file is in /etc/agent-config.toml, but when starting the kata-agent we specify this with the --config option e.g https://github.com/confidential-containers/cloud-api-adaptor/blob/816b76ae831b7c9e13cdbfc4c56f7c652f6f590a/podvm/files/etc/systemd/system/kata-agent.service#L8C8-L8C9

As peer pods can't set the kernel_params, we need to add some ability to read from the agent-config file if we are in a peer pod environment. For 0.8 we could hard-code search for /peerpod/daemon.json existence (to see if we are in a peer-pod) and if so read aa_kbc_params = "<val>" from /etc/agent-config.toml, then I think that should work correctly. I wonder if the best solution longer-term, rather than hard-coding is to add a --config option to CDH to specify the config.toml path to read it from like the kata-agent does, but I'm aware that policy might be replacing some of the functionality, so I don't want us to invest too much in that before then.

For more discussion see confidential-containers/cloud-api-adaptor#1424

[fitzthum]

Yeah, I think we are going to need a significant rethink of our configuration setup for v0.9.0 or v0.10.0. I think we want to move away from the command line as much as possible, but there are some subtleties. In the past the AA never read the command line directly. Instead the Kata Agent was responsible for calling it with the correct parameters. It may have been an oversight to diverge from that when we introduced the CDH.

[Xynnn007]

Yeah, I think we are going to need a significant rethink of our configuration setup for v0.9.0 or v0.10.0. I think we want to move away from the command line as much as possible, but there are some subtleties. In the past the AA never read the command line directly. Instead the Kata Agent was responsible for calling it with the correct parameters. It may have been an oversight to diverge from that when we introduced the CDH.

Yes, agree to rethink of our way to deliver configurations like aa_kbc_params. From my point of view we should get rid of deliver that from the kernel commandline. The benefits are:

• we do not need to add hardcoded items to verifier drivers to parse specific keys in kernel cmdline like aa_kbc_params= t o attestation service side every time a new configuration item is needed.
• For SNP any change to kernel cmdline will change the measurement.

From the peer pods, I found a new way to get the configurations from the metadata service of the CSP https://github.com/confidential-containers/cloud-api-adaptor/blob/main/cmd/process-user-data/provision.go#L132. The function is a systemd service named process-user-data running before kata-agent.

I also once shared an idea on kata-containers/kata-containers#7669 (comment) that to transfer the configurations via kata-agent API from the host.

Both ways protect the integrity of the configurations by binding to HOSTDATA field:

• peer pods get the peer pods specific configurations from metadata service using process-user-data service, and put to /peerpod/daemon.json
• we can impl: CoCo receives that from kata-agent's one-time api, and put to like /run/confidential-containers/configuration.json, or also the metadata service of CSP.

After that, kata-agent, confidential data hub and attestation-agent, etc will not need to parse aa_kbc_params, kbs certs.. from the kernel commandline.

Sorry that the thoughts are a little messy.

[fitzthum]

Yeah, I can think of three ways to pass the configurations.

1. Add a config API to the agent and put the hash of the config in the hostdata (won't work on every platform)
2. Add a config API and sign the configuration (with what key?)
3. Push the config from the KBS (can't be used to configure the CDH)

We could potentially use multiple different approaches for different platforms and deployment types.

If we go with the first solution we should work on standardizing some format for things that are measured by the host data. It seems like there are going to be a number of different uses for hostdata so we should have some kind of manifest that describes what is there.

I also think we should make sure that we have a way to provide measured and unmeasured configurations. This might add some complexity to whatever implementation we go with, but it would help reduce the complexity of verification.

[Xynnn007]

Agreed with the summary.

For TDX/SNP/SGX/CCA we can go way 1. For SEV-ES we can go way 3 (Yes pre-attestation helps to ensure the authcity of the KBS).

Also, if SEV-ES can support vtpm, it can also go way 1.

If we go with the first solution we should work on standardizing some format for things that are measured by the host data. It seems like there are going to be a number of different uses for hostdata so we should have some kind of manifest that describes what is there.

Yes. It would be easy if all init data are files, e.g. OPA policy, configuration of CDH, ...
Then we can have a map from path-in-the-guest onto file-content. For example

{
    "/etc/policy.rego": "<base64-encoded-content>",
    "/etc/cdh-conf.json": "<base64-encoded-content>",
    ...
}

We can reorder the keys of the json in dictionarial order for consistance. Calculate the digest and input the digest as HOSTDATA. Some tools to calculate the HOSTDATA should be provided for the tenants to calculate that ahead of deployment, and then they configure the Attestation Service OPA policy to match them. cc @arronwy @jialez0

I also think we should make sure that we have a way to provide measured and unmeasured configurations. This might add some complexity to whatever implementation we go with, but it would help reduce the complexity of verification.

My first impression is that all configurations will influence the behavior of the guest environment in a way, thus we do not need unmeasured configurations in productive env. However we can have one for debugging & tests. Let's go and see.

[fitzthum]

SEV-ES will most likely never support the first option. Not sure about s390x. I think we're going to need some backup options.

Let's continue the discussion of what needs to be measured and what doesn't over on your PR. One thing I do want to mention about configuration is we should think about where it is generated. For example, some configurations should probably be set via Kubernetes annotation. If we have a configuration file it will probably need to be at least partly generated on the fly.