Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The SCC specified in a DevWorkspace attributes is not used (an SCC with higher priority is picked) #884

Closed
l0rd opened this issue Jul 4, 2022 · 6 comments
Assignees
Milestone

Comments

@l0rd
Copy link
Collaborator

l0rd commented Jul 4, 2022

Description

EDIT
1. the DWO already exposes a mechanism to specify Pods SecurityContext (with the DWOC). So this issue has been closed and a corresponding Eclipse Che issue has been created.
2. This issue has been associated to the PR #954 that changes the way the workspace SA is granted the privileges to use an SCC (the problem was mentioned in this issue comment).

Following the instructions from this PR it's possible to run workspace Pod using a specific SCC.

That works fine until the workspace service account doesn't have any other SCC with same/higher priority. From OCP documenation

Admission uses the following approach to create the final security context for the pod:

  1. Retrieve all SCCs available for use.
  2. Generate field values for security context settings that were not specified on the request.
  3. Validate the final settings against the available constraints.
    (...)
    Security context constraints (SCCs) have a priority field that affects the ordering when attempting to validate a request by the admission controller. A higher priority SCC is moved to the front of the set when sorting. When the complete set of available SCCs are determined they are ordered by:
  4. Highest priority first, nil is considered a 0 priority
  5. If priorities are equal, the SCCs will be sorted from most restrictive to least restrictive
  6. If both priorities and restrictions are equal the SCCs will be sorted by name
    By default, the anyuid SCC granted to cluster administrators is given priority in their SCC set. This allows cluster administrators to run pods as any user by without specifying a RunAsUser on the pod’s SecurityContext. The administrator may still specify a RunAsUser if they wish.

How To Reproduce

  1. start a new cluster-bot cluster and install the devworkspace-operator
  2. follow the instructions in the PR
  3. create an SCC (with same or higher priority and with the system:authenticated group) and retry to start the workspace.
  4. verify that the Pod SCC is restricted-runasuser and not container-build

Expected behavior

Adding an SCC to workspaces SA is not enough, we need to explicitly list the required capabilities in the workspace Pod spec. I am not sure what's the best approach:

  1. automatically extracting the capabilities from the SCC
  2. requesting users to specify the capabilities in the DevWorkspace (with the new spec.pods?)

Additional context

These slides and presentation have great SCC details.

@AObuchow
Copy link
Collaborator

AObuchow commented Oct 12, 2022

@amisevsk and I have been looking into this issue further, and here is a summary of our current findings:

There are 2 potential workarounds that we've identified: increasing the priority of the container-build SCC and using the pod-override attribute.

Increasing the container-build SCC priority

Increasing the container-build SCC priority to be higher than the issue reproducer SCC will cause the workspace deployment to use the container-build SCC. In my testing, I modified the container-build SCC priority from 10 -> 30and it was used in the workspace pod deployment.

However, this solution may not be ideal for security reasons. As described by Angel:

Requiring a higher priority SCC for container-build is a bad bet, since it would mean that all pods started by Che users end up using the higher privilege SCC even when they don't intend to

Using the pod-override attribute

@amisevsk reported that applying any security context that isn't available in restricted-runasuser SCC causes the container-build SCC to be used instead. For example, specifying a security context with the runAsUser field set to 1000 (which is the UID used for buildah) will cause the container-build SCC to be used, as the restricted-runasuser specifies that a range of UID's must be given.
e.g.

  template:
    attributes:
      controller.devfile.io/scc: container-build
      pod-overrides:
        spec:
          securityContext:
            runAsUser: 1000

This does require some manual investigation in finding out which securityContext options will cause the container-build SCC to be used instead of the restricted-runasuser SCC.

Issues with mapping Security Context Constraints to Security Contexts

I tried @l0rd first suggested approach:

automatically extracting the capabilities from the SCC (and applying them to the workspace pod spec container's security context)

Unfortunately, there is no clear way to programmatically apply the fields from an SCC to a Security Context in order to force the desired SCC to be used for a workspace deployment. This is mainly due to the fact that SCCs constrain what you can use in a securityContext, but generally don't prescribe what you must use (thank you @amisevsk for this nice quote 😎).

Furthermore, the diff of the container-build and restricted-runasuser SCCs are actually fields that cannot be easily mapped onto a Security Context.
(Excuse the fact that below is an image and not text - was having trouble getting nice formatting for a side-by-side diff)
image

It is possible that extracting capabilities from the SCC is a potential solution, however I currently haven't figured out a way to go about this.

@l0rd
Copy link
Collaborator Author

l0rd commented Oct 13, 2022

Increasing the container-build SCC priority

Increasing the priority of the container-build SCC has side effects on other workloads and has been reported by some users as not possible on some real world clusters. So unfortunately that's not a solution and that's why this problem is urgent.

Using the pod-override attribute

This looks the best approach in my opinion. We just introduced the new attribute devEnvironments.disableContainerBuildCapabilities. When this is set to false, Che dashboard injects the attribute controller.devfile.io/scc: container-build. Maybe this issue can be addressed too by Che dashboard injecting the pod-overrides attribute too.

One comment on your proposal, rather than runAsUser I would explicitly add the capabilities rather than specifying runAsUser (does container-build SCC allows that?):

      pod-overrides:
        spec:
          securityContext:
-            runAsUser: 1000
+            capabilities:
+              add: [ "SETGID", "SETUID" ]

but I am wondering if the capabilities should not be set in the container(s) securityContext (rather than the pod) and if we should also explicitly specify allowPrivilegeEscalation: true in the Pod securityContext.

Issues with mapping Security Context Constraints to Security Contexts

I agree. That doesn't look like a good solution.

@l0rd
Copy link
Collaborator Author

l0rd commented Oct 13, 2022

When looking at this problem we also noticed that the container-build SCC is modified every time a new workspace is created: the workspace SA is added in the scc users list:

$ oc get scc container-build -o json | jq '.users'
[
  "system:serviceaccount:janedoe-devspaces:workspace684ea583240e4e80-sa"
]

This is not ideal as there may be some policies that prevent editing the SCC spec. Wouldn't it be possible to create a rolebinding instead?

@l0rd
Copy link
Collaborator Author

l0rd commented Oct 13, 2022

I have created eclipse-che/che#21770 that makes che-operator responsible to set the DevWorkspace config that adds the overrides. @amisevsk @AObuchow please have a look.

@amisevsk
Copy link
Collaborator

Creating a role/rolebinding may make sense, I will need to look into it.

@amisevsk
Copy link
Collaborator

amisevsk commented Nov 3, 2022

To update:

PR #954 updates how SCCs are handled to create a role/rolebinding rather than updating the SCC to add serviceaccounts. This PR did not get reviewed in time for DWO 0.17.0 and has a breaking change in that DWO would require the use permission in addition to get and update. As a result, this PR will be merged and available in DWO 0.18.0.

The conclusion regarding SCC priority is that this is expected behavior; the correct procedure for using an SCC with DevWorkspaces should be documented as

  1. Create an SCC for use with DevWorkspaces (e.g. container-build). Priority is unimportant, and can be set to 0
  2. Grant DWO permissions to get, update, use the SCC in question
  3. Add the controller.devfile.io/scc: <created SCC from step 1> attribute to DevWorkspaces that intend to use the SCC
  4. Configure the pod/container securityContext to use privileges granted by the custom SCC.
    • If all workspaces are intended to use the SCC, this can be configured via DWOC fields
    • Otherwise, it's possible to use pod-overrides and container-overrides attributes to override securityContext for specific workspaces

Note that SCC resolution works by finding the least privileged SCC that satisfies the securityContext for a pod. Often, it is sufficient to specify only one of the required fields (e.g. capabilities) to trigger the use of the intended SCC.

I'm closing this issue as we do not currently intend to automate the process of extracting a suitable securityContext from a given SCC to automate step 4 above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants