Ability to run "docker in docker"

[edeandrea]

Is your enhancement related to a problem? Please describe.

I'm working with Quarkus applications. Quarkus has this feature called DevServices. If my application requires a database, a Kafka broker, a Keycloak instance, or an Apicurio registry, when I run or test my Quarkus application (i.e. inner-loop development cycle) Quarkus can automatically spin up one of these containers for me and then shut it down when I shut down my application. Under the covers, Quarkus uses Testcontainers to do this.

This problem I'm having extends past Quarkus applications as well. Using Testcontainers directly (say in any other kind of Java application) I can satisfy many external dependencies that my application might need when running its tests (again, inner-loop development).

In Eclipse Che, this is impossible because there is no container runtime exposed to the workspace. I am not at all able to execute my test suite inside Eclipse Che.

Describe the solution you'd like

Ideally I'd like to just run mvn verify in my source tree (for a Java project) and have things just work. If my code uses Testcontainers, let Testcontainers do its "thing". If Quarkus, let Quarkus do its "thing".

Describe alternatives you've considered

Others have run into similar issues with things like Tekton, where as a step in the pipeline run they need to run mvn verify and have Testcontainers do its thing. They've solved that by using the docker-dind (docker-in-docker) image as a sidecar container and sharing the volume between the sidecar & "main" containers (see this great article that was written by someone who got this working in Tekton).

I tried to apply that same pattern to a CRW workspace using a sidecar image, mounting volumes, etc, but it doesn't work, probably because of privileged access/etc.

WIthout something like this it really is impossible to use Testcontainers to run my test suite or to even run a Quarkus application "locally" (inner-loop) within a CRW workspace.

[ibuziuk]

@l0rd @benoitf adding devex team
AFAIK we have not ever tried the case related to https://www.testcontainers.org/

[edeandrea]

Its VERY common amongst Java developers

[benoitf]

the biggest issue is that it requires privileged mode

[edeandrea]

It can be done, as my link to the article shows for doing it with Tekton. Not having this kind of thing, though, is going to make Che unusable for inner-loop development in the Java ecosystem, especially as we are trying to drive adoption of Quarkus. I'm not able to build workshops/demos using Quarkus on CRW to showcase features which differentiates it from other Java frameworks.

Also, think about when we build workshops for the field to deliver to customers (or as part of organized marketing events) - we heavily rely on CRW because it eliminates the local machine setup headaches. As Quarkus evolves and matures, CRW no longer is something we can use for showcasing how awesome Quarkus is and the developer productivity features it provides for the inner-loop development cycle.

[brootware]

This would be a great feature for writing dockerfiles also in Che.

[edeandrea]

What about running podman instead of docker in a container (testcontainers/testcontainers-java#2088 or containers/podman#7927)?

[l0rd]

@brootware @edeandrea we understand the value of this feature. But allowing arbitrary developers to run a pod with

securityContext:
    privileged: true

as in the Tekton example will fail on real world kubernetes clusters (c.f. https://snyk.io/blog/10-kubernetes-security-context-settings-you-should-understand/). For instance it won't work on Red Hat developer sandbox.

[edeandrea]

Circling back to this thread. If you take a look at quarkusio/quarkus#19132 (comment) you'll notice that this is possible on other platforms as well (i.e. gitpod, GitHub actions, etc). Now I'm not sure how they're accomplishing it, but they are exposing some way where the application can detect a docker daemon running.

This is the ultimate goal here and is something we're struggling to do with Eclipse Che.

[edeandrea]

FYI this is what I was trying to do...

apiVersion: 1.0.0
metadata:
  generateName: quarkus-devservices-
components:
  - id: redhat/quarkus-java11/latest
    type: chePlugin
  - mountSources: true
    endpoints:
      - name: app
        port: 8080
    memoryLimit: 1Gi
    type: dockerimage
    volumes:
      - name: m2
        containerPath: /home/jboss/.m2
      - name: dind-socket
        containerPath: /var/run
    image: 'registry.redhat.io/codeready-workspaces/plugin-java11-rhel8:latest'
    alias: app
    env:
      - value: '-XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Djava.security.egd=file:/dev/./urandom -Duser.home=/home/jboss'
        name: JAVA_OPTS
      - value: $(JAVA_OPTS)
        name: MAVEN_OPTS
  - mountSources: true
    alias: docker
    memoryLimit: 1Gi
    type: dockerimage
    image: 'docker:dind-rootless'
    volumes:
      - name: dind-storage
        containerPath: /var/lib/docker
      - name: dind-socker
        containerPath: /var/run
commands:
  - name: Attach remote debugger
    actions:
      - referenceContent: |
          {
            "version": "0.2.0",
            "configurations": [
              {
                "type": "java",
                "request": "attach",
                "name": "Attach to Remote Quarkus App",
                "hostName": "localhost",
                "port": 5005
              }
            ]
          }
        type: vscode-launch

[maxandersen]

for what it is worth testcontainers and thus quarkus devservices just works in gitpod.io and github codespaces. Here docker deamon is accessible.

[benoitf]

the securityContext: privileged is not granted on OpenShift hosted solutions so for example it can't work on workspaces.openshift.com
The limitation is not coming from Eclipse Che

[edeandrea]

I guess my question is - is this possible without the securityContext: privileged ? Is there a way to get a container runtime available in a container (or as a sidecar) without the need for privileged?

[l0rd]

@edeandrea not when we tested ~1 year ago. But it looks like things have changed and we need to have a look again (c.f. this issue and related OpenShift enhancement proposal.

[l0rd]

With this merged and with the switch to DevWorkspace, it should be possible to docker build (buildah and docker should be part of universal developer image) from within a Che workspace but:

• the DevWorkspace Operator SA must explicitly be granted get and update privileges for anyuid SCC
• the user creating the DevWorkspace must be granted the use privilege on the anyuid SCC
• the Devfile defining the workspace has attribute controller.devfile.io/scc: anyuid
• only works on OpenShift

(more details can be found in the PR)

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[l0rd]

/remove-lifecycle stale

[cgruver]

A lot of progress has been made recently on this one. It is now possible to use podman build or buildah to create a container image.

However, the ability to run a container still seems elusive. @l0rd Do you know if there is active work to allow for running containers within a Che workspace? I'm assuming that since you assigned it to yourself, there is some activity. ;-)

[l0rd]

Yes, we are working on it. I have been able to use kubedock to run “docker run” from within a workspace on OpenShift. We should add kubedock in the default developer image (UDI) this sprint.

[cgruver]

Thanks! In the mean time @l0rd, do you have a link to some POC code? I'm running Che off of the next stream in my lab. I'd love to get a jump on testing out Quarkus dev services and other use cases.

I assume that by kubedock you are referring to: https://github.com/joyrex2001/kubedock?

[l0rd]

Yes, that's the kubedock I am referring too. Yes I have PoC. Will share the link in a few minutes.

[l0rd]

The sample repo is this one: https://github.com/l0rd/micronaut-sample
Once the worksapce is up you need to start kubedock and then you are able to execute docker commands. I have also recorded a short video.

Note that:

• kubedock can be started as the image entrypoint but we cannot because of this nasty bug
• an error is shown when you do docker run but that's normal (a bug kubedock side probably)
• podman run is not supported yet

[cgruver]

@l0rd @edeandrea Quarkus Dev Services appears to work with some tweaking to the RBAC on the eclipse che namespace created for the user.

I'm going to attempt to refine it. I brute forced it by giving my user admin on the namespace:

oc policy add-role-to-user admin cgruver -n cgruver-che

But I don't think it needs all of that, just the ability to create Jobs and maybe a few other things not currently included in the applied RBAC.

I successfully ran this example: https://github.com/joyrex2001/kubedock/tree/master/examples/quarkus-devservices

My devfile is here: https://github.com/cgruver/che-quarkus-demo

It needs the quarkus ImageStream and BuildConfig from here: https://raw.githubusercontent.com/cgruver/eclipse-che-developer-images/main/install.yaml

I'll get all of this consolidated into a cohesive demo soon.

[cgruver]

This may not be all of the privileges needed for other kubedock functions, but it works for Quarkus dev services:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: quarkus-dev-services
rules:
- apiGroups:
  - batch
  resources:
  - jobs
  - jobs/status
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - pods/portforward
  verbs:
  - get
  - list
  - watch
  - create
  - delete
  - deletecollection
  - patch
  - update

oc policy add-role-to-user quarkus-dev-services <user-id> -n <user's-che-namespace>

[l0rd]

@cgruver fyi it's possible to specify an SCC to associate to every developer account. That's with CheCluster field .spec.devEnvironments.containerBuildConfiguration.openShiftSecurityContextConstraint.

[cgruver]

@l0rd I don't believe that an SCC is what we need for this, is it?

Would it make sense to just incorporate the above in the eclipse-che-cheworkspaces-clusterrole ClusterRole definition?

The ability to port-forward from a running pod in the namespace to the workspace pod seems like a reasonable use case.

I'm not sure why kubedock needed list for Job, but it was throwing errors about that while creating the pods for the Quarkus tests. It's likely related to other capabilities of kubedock. I've barely scratched the surface I think. The ability to port-forward to the running shell in the workspace is the critical capability for this particular use case with Quarkus dev services.