Make runc's proc mount work again

[csweichel]

Bug description

A recent change in runc broke procfs mounts in Gitpod workspaces. Specifically the way proc is mounted now (see here) seems to break.

This might be a matter of "faking" the check in workspacekit.

Steps to reproduce

Run anything with a recent runc, e.g. current buildkit or containerd version.

Expected behavior

The proc mount should work just fine.

Example repository

No response

Anything else?

No response

[csweichel]

How to reproduce easily

sudo curl -o $(which runc) -L https://github.com/opencontainers/runc/releases/download/v1.0.1/runc.amd64
docker run --rm -it alpine:latest

[csweichel]

/assign

[jemand771]

just to be sure I don't open a duplicate of this - is this error the same or caused by something else?

$ docker run --rm hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete 
Digest: sha256:776b0895d5e2fcd5e80bcdd607adc45461ba11143ef3df531174bf00679f43fe
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "proc" to rootfs at "/proc" caused: mount through procfd: operation not permitted: unknown.

additional info:

$ docker --version
Docker version 19.03.15, build 99e3ed8919
$ containerd --version
containerd containerd.io 1.4.9 e25210fe30a0a703442421b0f60afac609f950a3
$ runc --version
runc version 1.0.1
commit: v1.0.1-0-g4144b63
spec: 1.0.2-dev
go: go1.15.14
libseccomp: 2.4.3

[csweichel]

this is indeed the same issue

[jemand771]

okay, thanks. looking forward to a fix 👀

[patajones]

the same problem here

$ docker run ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
16ec32c2132b: Pull complete 
Digest: sha256:82becede498899ec668628e7cb0ad87b6e1c371cb8a1e597d83a47fac21d6af3
Status: Downloaded newer image for ubuntu:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "proc" to rootfs at "/proc" caused: mount through procfd: operation not permitted: unknown.
ERRO[0005] error waiting for container: context canceled 

Using container image gitpod/workspace-full

Docker version:

Client: Docker Engine - Community
 Version:           19.03.15
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        99e3ed8919
 Built:             Sat Jan 30 03:17:01 2021
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.15
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       99e3ed8919
  Built:            Sat Jan 30 03:15:30 2021
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 gitpod:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Something I noticed, when I open the demo project https://gitpod.io/#https://github.com/gitpod-io/spring-petclinic , its work. Looking docker version the differences are:

containerd 1.4.4
gitpod Version 1.0.0-rc93

[patajones]

is there a workaround for this problem?

To reproduce the problem, open https://gitpod.io/#https://github.com/patajones/gitpod-test and execute docker run something

[gclayburg]

I just ran into this issue on a gitpod instance. I also noticed that things worked fine when I opened another project. The one that fails has a newer runc installed:

gitpod /workspace/policyStart $ /usr/bin/runc --version
runc version 1.0.1
commit: v1.0.1-0-g4144b63
spec: 1.0.2-dev
go: go1.15.14
libseccomp: 2.4.3

The gitpod instance where docker works uses 1.0.0-rc93:

gitpod /workspace/upbanner $ runc --version
runc version 1.0.0-rc93
commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
spec: 1.0.2-dev
go: go1.13.15
libseccomp: 2.4.3

@patajones I was able to work around this issue by downloading and installing an older version of runc. I have no idea if this would cause other problems though. So far, docker is working for me now.

$ curl -o olderrunc -L https://github.com/opencontainers/runc/releases/download/v1.0.0-rc93/runc.amd64
$ chmod 755 olderrunc
$ sudo cp -p /usr/bin/runc runc.1.0.1.orig
$ sudo cp olderrunc /usr/bin/runc

gitpod /workspace/policyStart $ runc --version
runc version 1.0.0-rc93
spec: 1.0.2-dev
go: go1.14.14
libseccomp: 2.5.1

[ChristianHuff-DEV]

@gclayburg Thanks for sharing your workaround. It works for me too.

[WesleyMConner]

@gclayburg ... A brute-force roll back in my current gitPod instance works for me as well.
@csweichel - Is Gitpod likely to roll back to the older runc (for new instances) while work on a fix continues?

[patajones]

Pull Request #5139 didn't fix the problem.
Thanks @gclayburg for workaround

[jemand771]

@patajones this issue still has the "awaiting deployment" label, so I guess it will take a short while for the fix to become active
https://github.com/orgs/gitpod-io/projects/3#card-66515391

[ArthurSens]

We've just deployed this to production, but the traffic is shifted slowly to make sure we don't put too much load in the new cluster.

If you open a workspace now and see eu14 or us14 in its URL, this should be fixed. If you end up on "15", you can close the workspace and try to open another one until you end up on "14".

At this exact moment, the chances to end up in each cluster is 50%-50%, but we should have 100% on the cluster with the fix in the next few hours

[paul121]

This happening to me with v1.0.0-rc93. Seems to be affecting all containers, below is the output after opening https://github.com/shaal/DrupalPod

Starting drupalpod... 
Failed to start drupalpod: failed to StartContainer: API error (500): OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: rootfs_linux.go:60: mounting "proc" to rootfs at "/proc" caused: operation not permitted: unknown 

gitpod /workspace/DrupalPod $ runc --version
runc version 1.0.0-rc93
commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
spec: 1.0.2-dev
go: go1.13.15
libseccomp: 2.4.3

[wolfram-laube]

Experiencing

docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: rootfs_linux.go:60: mounting "proc" to rootfs at "/proc" caused: operation not permitted: unknown.

also after having applied @gclayburg 's workaround and with runc version:

runc version 1.0.0-rc93
spec: 1.0.2-dev
go: go1.14.14
libseccomp: 2.5.1

[wolfram-laube]

I also tried starting new workspaces and hope to get some eu14 or us14 but chances are a little less than 50%, it seems...

[ghuntley]

Follow #5171 for status updates and implementation of the corrective fix.