Container build support doesn't work after upgrade from 7.58.0 to 7.59.0

[cgruver]

Describe the bug

Che version 7.59 seems to have broken rootless podman on OpenShift

OpenShift - OKD 4.12 SCOS

Che version

7.59@latest

Steps to reproduce

1. Install Che with chectl server:deploy -p openshift
2. Enable container builds:
   oc patch CheCluster eclipse-che -n eclipse-che --type merge --patch '{"spec":{"devEnvironments":{"disableContainerBuildCapabilities":false}}}'
3. Create a workspace with the empty project sample
4. Open a terminal in che-code
5. Execute podman images
6. Observe error:

   WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers 
   ERRO[0000] running /usr/bin/newuidmap 289 0 1000700000 1 1 10000 65536: newuidmap: write to uid_map failed: Operation not permitted 
   Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

Expected behavior

Successful empty list of images

Runtime

OpenShift

Screenshots

No response

Installation method

chectl/latest

Environment

macOS

Eclipse Che Logs

No response

Release Notes Text

A regression in the support for building containers has been fixed in this release.

[cgruver]

Possible hint. I updated to the Next channel and now I get an informative error on creating a workspace:

Failed to start the workspace workspace-dev, reason: Failed to update workspace 'workspace-dev'. Unable to patch devworkspace: admission webhook "mutate.devworkspace-controller.svc" denied the request: user is not permitted use the container-build SCC

[ibuziuk]

@cgruver thank you for reporting the issue.
Could you please clarify which version of Dev Workspace Operator is used on the cluster? Basically, with the latest 7.59.0 and DWO 0.18.0 podman builds should work fine - eclipse-che/che-operator#1558

[cgruver]

Output of oc get csv:

devworkspace-operator.v0.18.1   DevWorkspace Operator   0.18.1            devworkspace-operator.v0.18.0   Succeeded

[l0rd]

I can reproduce that on our internal (dogfooding) instance.

Looking at this issue I have found a few other problems:

1. the UDI tag used when starting an empty workspace is ubi8-38da5c2 which is 6 months old (!)
2. the empty workspace DevWorkspace includes VS Code metadata although those are already defined as contribution in the VS Code metadata

[cgruver]

Let me know if you come up with a work around that I can apply to my cluster. Or... if it's possible to reinstall 7.58.

I downgraded chectl back to 7.58, but it still installs the latest from the stable channel in from Operator Hub.

[cgruver]

Work around:

chectl server:deploy --starting-csv=eclipse-che.v7.58.0 -p openshift

This will install v7.58.0 with a manual update strategy and avoid v7.59.0 for now.

[amisevsk]

I've reproduced this on OpenShift 4.11 and come out even more confused than before after testing.

Testing 7.58.0 vs 7.59.0, I've found

• In both versions, the workspace starts up, there is no issue 'using' the container-build scc. The same version of DWO is used behind the scenes in both versions when installing in a new cluster
• In both versions, the devworkspace, scc, and workspace pod are identical, with the only difference being a different image for the che-code-injector component and the addition of

      allowPrivilegeEscalation: false
      capabilities:
        add:
          - SETGID
          - SETUID

  to the pod security context
• After testing update from 7.58.0 to 7.59.0 twice, I saw
  • First time: podman images succeeds in 7.59.0 after updating
  • Second time: podman images fails as expected, but old che-code-injector image is used (is registry not being updated?)

I haven't tested for Che next, but that error is even stranger. The error

Failed to start the workspace workspace-dev, reason: Failed to update workspace 'workspace-dev'. Unable to patch devworkspace: admission webhook "mutate.devworkspace-controller.svc" denied the request: user is not permitted use the container-build SCC

is reported when the user does not have permissions to use the requested SCC, which is not occuring in either 7.58.0 nor 7.59.0

[amisevsk]

Testing a little further, it appears to be due to Che 7.59.0 adding

  config:
    workspace:
      containerSecurityContext:
        allowPrivilegeEscalation: false

to the default Pod security context via its custom DevWorkspaceOperatorConfig.

Is there a reason we use false here? The SCC Che defines includes

allowPrivilegeEscalation: true

[amisevsk]

PR eclipse-che/che-operator#1576 sets allowPrivilegeEscalation: false in the DWOC.

[l0rd]

We should set it to true.

[cgruver]

Very ugly workaround... that will at least let you work with containers in 7.59 for now:

oc patch dwoc devworkspace-config -n eclipse-che --type merge --patch '{"config":{"workspace":{"containerSecurityContext":{"allowPrivilegeEscalation":true}}}}'

You will have to rerun it immediately after creating a new workspace because the operator will revert it to false every time it touches the DWOC.

[cgruver]

@l0rd @amisevsk Let me know when you have a new build on the Next channel. I'll update one of my clusters to that version.

[amisevsk]

@cgruver I've opened PR eclipse-che/che-operator#1596; once merged should be pushed to the next channel fairly quickly (though I still haven't looked into the issue you encountered with the next channel, so there may be further issues). We should also backport this PR to the 7.59.x branch to fix the bug for all users.