✨ add explicit securitycontexts to controllers

[tuminoid]

Set explicit, secure securityContexts for the controller manager deployment and containers instead of relying on defaults or fallbacks.

Only actual change here is enabling runtimeDefault seccompPolicy, instead of running as Unconfined.
https://kubernetes.io/docs/tutorials/security/seccomp/

Also, reindent poorly indented command block.

[k8s-ci-robot]

Welcome @tuminoid!

It looks like this is your first PR to kubernetes-sigs/cluster-api 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @tuminoid. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tuminoid]

Hello maintainers! I'd like to hear your opinion on adding explicit securityContexts to CAPI controller manager.

• Adding them to manifest vs leaving it for cluster admins and cluster/namespace wide policies?
• Directly to the manager manifest vs creating kustomization?
• Adding seccompProfile, changing unconfined -> runtimeDefault?-

[sbueringer]

Hello maintainers! I'd like to hear your opinion on adding explicit securityContexts to CAPI controller manager.

• Adding them to manifest vs leaving it for cluster admins and cluster/namespace wide policies?
• Directly to the manager manifest vs creating kustomization?
• Adding seccompProfile, changing unconfined -> runtimeDefault?-

1. I would add them to our manifests like you did
2. I would add them to the manager as you did - because it's simpler
3. I'm fine with adding seccompProfile assuming it will work on all Kubernetes clusters (I'm not really familiar with the seccomp feature in Kubernetes)

P.S. I assume you wanted to clarify those points first. Eventually we should modify the manager manifests of CABPK, KCP and CAPD in this PR as well (CAPD won't work as non-root because of the docker socket mount, but let's explicitly add what we can)

[tuminoid]

Hello maintainers! I'd like to hear your opinion on adding explicit securityContexts to CAPI controller manager.

• Adding them to manifest vs leaving it for cluster admins and cluster/namespace wide policies?
• Directly to the manager manifest vs creating kustomization?
• Adding seccompProfile, changing unconfined -> runtimeDefault?-

1. I would add them to our manifests like you did
2. I would add them to the manager as you did - because it's simpler
3. I'm fine with adding seccompProfile assuming it will work on all Kubernetes clusters (I'm not really familiar with the seccomp feature in Kubernetes)

Thanks for the reply! We had a discussion in CAPM3 that we want to align with you (CAPI) first on this, and then replicate the approach to CAPM3, BMO, IPAM and elsewhere where necessary.

As for you question about seccompProfiles, the link in the PR description describes them. RuntimeDefault is detailed here: https://kubernetes.io/docs/tutorials/security/seccomp/#create-pod-that-uses-the-container-runtime-default-seccomp-profile

In short, all (or almost all) container runtimes come with a default profile that limits some of the syscalls from the containers. They're typically not used by non-root workloads, so enabling it changes nothing. In case container calls restricted syscall, it is prevented and app will crash. In our testing, I've enabled it for CAPI, CAPM3, BMO, IPAM without any issues.

P.S. I assume you wanted to clarify those points first. Eventually we should modify the manager manifests of CABPK, KCP and CAPD in this PR as well (CAPD won't work as non-root because of the docker socket mount, but let's explicitly add what we can)

I can amend the PR for sure. Would you ok-to-test the PR so I can verify CAPI tests pass before I do that?

[sbueringer]

/ok-to-test
(sorry missed that)

[sbueringer]

In short, all (or almost all) container runtimes come with a default profile that limits some of the syscalls from the containers. They're typically not used by non-root workloads, so enabling it changes nothing. In case container calls restricted syscall, it is prevented and app will crash. In our testing, I've enabled it for CAPI, CAPM3, BMO, IPAM without any issues.

The interesting question is what happens if the container runtime doesn't come with a default profile.

If I understand it correctly starting with Kubernetes 1.25 the SeccompDefault (https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) feature gate will be enabled per default which sets the default profile automatically.

So I think overall we could have the following problems:

• Does this also work with Kubernetes 1.20 (our minimum supported mgmt cluster Kubernetes version)
• Are there container runtimes that don't have a default profile

Given that starting with Kubernetes 1.25 Kubernetes takes care of setting the default seccomp profile automatically I would prefer not setting it in ClusterAPI to not run into issues in the edge cases. (but just my opinion, no objection if folks want to enable it and think it's safe)

[tuminoid]

In short, all (or almost all) container runtimes come with a default profile that limits some of the syscalls from the containers. They're typically not used by non-root workloads, so enabling it changes nothing. In case container calls restricted syscall, it is prevented and app will crash. In our testing, I've enabled it for CAPI, CAPM3, BMO, IPAM without any issues.

The interesting question is what happens if the container runtime doesn't come with a default profile.

If I understand it correctly starting with Kubernetes 1.25 the SeccompDefault (https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) feature gate will be enabled per default which sets the default profile automatically.

SeccompDefault feature gate enables kubelet to use --seccomp-default to enable RuntimeDefault for all workloads, but unless it is added to kubelet command line, feature gate is not doing anything.

You must also explicitly enable the defaulting behavior for each node where you want to use this with the corresponding --seccomp-default [command line flag](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet). Both have to be enabled simultaneously to use the feature.

So I think overall we could have the following problems:

* Does this also work with Kubernetes 1.20 (our minimum supported mgmt cluster Kubernetes version)

* Are there container runtimes that don't have a default profile

Given that starting with Kubernetes 1.25 Kubernetes takes care of setting the default seccomp profile automatically I would prefer not setting it in ClusterAPI to not run into issues in the edge cases. (but just my opinion, no objection if folks want to enable it and think it's safe)

• https://kubernetes.io/docs/tutorials/security/seccomp/ seccomp is supported as stable from k8s 1.19
• 1.25 does not have it as default, see above

If the container runtime doesn't have a default profile, then it'll be no different than Unconfined, as the profile filters syscalls, or depending on your runtime configuration, it may already be using runtime defaults.

https://github.com/cri-o/cri-o/blob/main/docs/crio.conf.5.md

• seccomp_profile="" Path to the seccomp.json profile which is used as the default seccomp profile for the runtime. If not specified, then the internal default seccomp profile will be used.
• seccomp_use_default_when_empty=true Changes the meaning of an empty seccomp profile. By default (and according to CRI spec), an empty profile means unconfined. This option tells CRI-O to treat an empty profile as the default profile, which might increase security.

https://docs.docker.com/engine/security/seccomp/ for Docker. Good list of syscalls filtered, showing most of them are namespaced, and not available anyways as we drop all capabilities, + few obsolete or non-namespaced calls.

containerd follows Docker profile: containerd/containerd#5924 and https://github.com/containerd/containerd/blob/f0a32c66dad1e9de716c9960af806105d691cd78/contrib/seccomp/seccomp_default.go#L51

and so on.

Of course, if you think seccomp is too risky to enable by default, I can leave it out of this PR.

[tuminoid]

/retest

[sbueringer]

Thx for the additional context! Sounds okay to me, but I would like to hear more opinions from others.

[bengentil]

Setting a security context in the manifest will break tilt live_update

This means for each code update, the pod won't be updated and tilt will report this error:

capi_control… │ - '<you local path>/cluster-api/.tiltbuild/bin/manager' --> '/manager'
capi_control… │ tar: can't remove old file manager: Permission denied

From what I understood from tilt-dev/tilt#3060,
the only way to make it work right now is to dynamically remove the security context in the Tiltfile.
I can try to propose a fix in that direction if you want.

[sbueringer]

@bengentil Nice catch, thx! Given how our tilt setup works I think that is something that we can do in our tilt-prepare binary (roughly here). But would be good to have verification that dropping it there works.

[bengentil]

@sbueringer you're right it's way easier and less error prone to implement it in tilt-prepare

I've successfully tested this fix: bengentil@d5eb23a

Don't know the process if it has to be in the same PR, but it should be merged before or at the same time ideally, feel free to take my commit in this branch if needed.

[sbueringer]

EDIT: I think it's okay to open this as a separate PR and merge it before. Especially given that this makes our tilt setup compatible with infra providers setting securityContext

[bengentil]

PR created: #7846

[tuminoid]

Thanks @bengentil for spotting the tilt issue and the PR to fix it! It'd be good to have it merged first.

I've updated the PR by splitting the indentation fix to separate commit and by adding securityContext's to KCP and CABPK. CAPD is privileged and none of the securityContexts make sense or have effect when privileged, so I did not touch it.

[fabriziopandini]

This is a nice change, I have modified the PR title so it stands out on release notes!
Can we document this change in upgrade notes for the providers as well?

• https://main.cluster-api.sigs.k8s.io/developer/providers/v1.3-to-v1.4.html#other --> Cluster API controllers are now using an explicit security context by default.
• https://main.cluster-api.sigs.k8s.io/developer/providers/v1.3-to-v1.4.html#suggested-changes-for-providers --> Providers should add an explicit security context to their controllers deployment, see ✨ add explicit securitycontexts to controllers #7831 for reference

Also, it will be great if we apply security policies to our test extension in https://github.com/kubernetes-sigs/cluster-api/blob/main/test/extension/config/default/manager.yaml (It should be pretty straightforward, but if I'm wrong this could be done also a follow-up PR)

[killianmuldoon]

I think lint will have to be retriggered by someone with the correct rights for the repo, but I think we should be happy to merge if an approver verifies that lint is working locally - not sure why it's stuck.

[tuminoid]

I think lint will have to be retriggered by someone with the correct rights for the repo, but I think we should be happy to merge if an approver verifies that lint is working locally - not sure why it's stuck.

Cool. It should not fail, as lint action is linting Go files, and this PR doesn't change any Go sources.

[sbueringer]

I had to "re-trigger" it. Fyi it was not the "new contributor" case where someone with write access has to click a button. This was the case where "somehow" GitHub just doesn't trigger the GitHub action.

I triggered the linter by essentially creating a "PR edited" event by adding a new empty line somewhere in the PR description...

Only had this 2-3 times up until now and it was always on dependabot PRs

[sbueringer]

/lgtm

/assign @fabriziopandini

[tuminoid]

Rebased due changelog conflict. Please re-review and let's get this merged. This already has had approved/lgtm from everyone required.

[sbueringer]

Thx!

/lgtm
/approve

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: d55651945044364f8715f476a681aa010d950fc7

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment