feat: follow KEP-2568 non-root enhancements

[nberlee]

Pull Request

WIP: works for new clusters, haven't tested upgrades (likely this breaks because of existing file permissions...?)
EDIT: tested and it just works, even with --preserve files are recreated with correct permission

Question: Should I include the seccomp yaml for the control-plane?

What? (description)

Follow KEP-2568 (https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2568-kubeadm-non-root-control-plane). Seperation UIDs, Adding GIDs, allowPriviledgeEscalation: false,drop unneeded capabilities and add seccomp CP manifest changes.

Seccomp is excluded as i think it has a dependency on #5293 ? Also I have no idea how to test that.

Why? (reasoning)

Mainly better compatibility (this KEP is scheduled to be implented for kubeadm 1.25) + security:

• allowPrivilegeEscalation: false: adds no_new_privs flag on the container process
• Capabilities: { Drop: [ "ALL" ] }: Capabilities are a more fine-grained permissions model, and all capabilities should be dropped from a pod, with only those required added back. (NET_BIND_SERVICE is needed for kube-apiserver as it has setcap flags)
• RunAsUser & RunAsGroup: Assigning an unique ID helps with limiting the file exposure of secrets

Acceptance

Please use the following checklist:

• you linked an issue (if applicable)
• you included tests (if applicable)
• you ran conformance (make conformance) (GPG identity fails)
• you formatted your code (make fmt)
• you linted your code (make lint)
• you generated documentation (make docs)
• you ran unit-tests (make unit-tests)

See make help for a description of the available targets.

[smira]

/ok-to-test

[nberlee]

So regarding my question in the PR description: Should I include the seccomp yaml for the control-plane?

Everything still works in QEMU when i add

							SeccompProfile: &v1.SeccompProfile{
								Type: v1.SeccompProfileTypeRuntimeDefault,
							},

As it is part of the KEP-2568, should I add it? I am asking as there is already an issue regarding this for all containers: #5293 and I am not sure if there is something on the OS level that still needs to be done, or why the issue is hold back...
If there is, I think it's actually bad to add this. If not, I am happy to add this to this PR or a new one.

[frezbo]

So regarding my question in the PR description: Should I include the seccomp yaml for the control-plane?

Everything still works in QEMU when i add

							SeccompProfile: &v1.SeccompProfile{
								Type: v1.SeccompProfileTypeRuntimeDefault,
							},

As it is part of the KEP-2568, should I add it? I am asking as there is already an issue regarding this for all containers: #5293 and I am not sure if there is something on the OS level that still needs to be done, or why the issue is hold back... If there is, I think it's actually bad to add this. If not, I am happy to add this to this PR or a new one.

I'll be looking into adding the seccomp support this week and see if there's anything extra needed.

[frezbo]

I guess you can add it, since kubelet will not enforce it as long as the feature-gate/cli flags are set, so I think it's okay to do this in this PR

[nberlee]

I guess you can add it, since kubelet will not enforce it as long as the feature-gate/cli flags are set, so I think it's okay to do this in this PR

done

[smira]

thanks!

yep, --preserve is fine as all the mounted files are created on tmpfs anyways, so they are thrown away with reboot.

[smira]

/m