Permissions in docker images are lost during gitpod restore

[shaal]

UPDATE:

This problem is actually not just about setuid, but about permissions in general.
Follow this comment #3174 (comment) for an even simpler steps to reproduce the issue.

---

Describe the bug

In a workspace using sudo docker-up and pulling a docker image, it has sudo with the correct setuid bit.
-rwsrwxrwx 1 root root 157192 Feb 2 2020 /usr/bin/sudo

But after timeout/restarting the workspace, the sudo in the image has the wrong setuid bit
-rwxr-xr-x 1 root root 157192 Feb 2 2020 /usr/bin/sudo

Steps to reproduce

I created a minimal setup to reproduce the behavior:
https://gitpod.io/#https://github.com/shaal/gitpod-restart-test

1. Start the workspace
2. The workspace will run sudo docker-up
3. The workspace will pull an image and display the image's /usr/bin/sudo permissions (ie. docker run -it --rm drud/ddev-webserver:v1.16.3 ls -l /usr/bin/sudo)
4. The permission should be -rwsrwxrwx
5. Using https://gitpod.io/workspaces - Stop the current workspace (alternatively you can wait for the workspace to timeout)
6. Start the same workspace again
7. Run docker run -it --rm drud/ddev-webserver:v1.16.3 ls -l /usr/bin/sudo
8. Now the permission is wrong -rwxr-xr-x, it's missing the setuid bit.

• Please note that the issue happens with any image.

Expected behavior

The setuid bit should stay the same after restarting a workspace.

Additional information

Example repository

https://github.com/shaal/gitpod-restart-test

[ArthurSens]

Thanks for the report @shaal!

Sorry if the answer is obvious, but how does changing the setuid bit affects your work when using Gitpod? I'm not saying it isn't a bug, just thinking about prioritization... 🤔

[shaal]

Sorry, I tried being specific and wanted to show the simplest way to reproduce the bug.

The way it affects my work, is that when a workspace times out, I can no longer work on that workspace again, until I destroy all docker containers.

You can see an example project here -
https://gitpod.io/#https://github.com/shaal/ddevenv

It opens one terminal with sudo docker-up, and another terminal with ddev start (pulling docker images to build the environment).
When starting the workspace everything works well.
Run gp url 8080, and open that URL in browser.
You would see the following:

Now stop the workspace (or wait for it to timeout)
Start the workspace again.

You'll see in the 3rd terminal, the command ddev start, failing with this error:

Starting ddevenv... 
Building ddev-ssh-agent 
Recreating ddev-ssh-agent ... done
 
Failed to start ddevenv: ddev-ssh-agent failed to become ready; debug with 'docker logs ddev-ssh-agent'; error: container /ddev-ssh-agent unhealthy:  

Running docker logs ddev-ssh-agent display this:

sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set
Creating a proxy socket...
Running socat UNIX-LISTEN:/tmp/.ssh-agent/proxy-socket,perm=0666,fork UNIX-CONNECT:/tmp/.ssh-agent/socket
Launching ssh-agent...
2021/02/11 18:53:12 socat[11] E bind(5, {AF=1 "/tmp/.ssh-agent/proxy-socket"}, 30): Permission denied
unix_listener: cannot bind to path /tmp/.ssh-agent/socket: Permission denied

[rfay]

The fundamental problem is that several of the images used for ddev have sudo in them, and of course sudo is not usable with the setuid bit reset inappropriately. I'm sure there are other utilities that would be hobbled with the setuid inappropriately reset, but sudo is the big one.

When a workspace is awakened, the setuid bit is no longer set on sudo inside these images. It should not have been changed.

And it doesn't matter if you re-pull the image. Same result.

[csweichel]

Thank you for reporting this.

This look like an issue with the content init during workspace startup. Apparently we're not preserving file modes correctly in some cases.

[rfay]

I'm obviously a novice to the codebase, but a common way to lose setuid bits and ownership information is to run tar x without root privileges. I wonder if that's happening here:

gitpod/components/content-service/pkg/archive/tar.go

Line 102 in 4d3947d

tarcmd := exec.Command("tar", "x")

[rfay]

This problem is actually not just about setuid, but about permissions in general.

In a docker image that had a directory with permissions 0777 (/etc/alternatives), after restore it has 0755 privileges.

@shaal maybe you can update the title to add "different permissions entirely"

[rfay]

I put together a repro case for the loss of permissions. It shows that both files and directories inside a docker image have their permissions changed (lost) during a gitpod restore.

Please see https://github.com/rfay/gitpod-bug-repro or go straight to https://gitpod.io/#https://github.com/rfay/gitpod-bug-repro - it's easy to follow and recreate, and shows at least 3 cases of lost permissions, not just setuid (as was the OP here) but also directory and file permissions of 777 get cut back to 755.

[aledbf]

@rfay thank you for the repository reproducing the issue.

[shaal]

@aledbf @csweichel is it possible to re-open this issue? I think the bug is not resolved yet.

The new Gitpod , does restore the workspace with the correct file permissions, but the owner of the files is 33333 instead of root.

I forked @rfay repo, and added a .gitpod.yml file that runs the docker command during prebuild.
https://github.com/shaal/gitpod-bug-repro

You can run the command manually, according to the Readme of that repo to see the same issue.

The result after workspace restore: (where file owner should be root and not 33333)

/var/tmp:
total 8
drwxrwxrwx 2 www-data 33333 4096 Feb 17 21:16 dir-with-perms
drwxr-xr-x 2    33333 33333 4096 Feb 17 21:28 filebits
/var/tmp/dir-with-perms:
total 0
/var/tmp/filebits:
total 0
-rwsr-xr-x 1 33333 33333 0 Feb 17 21:28 setuid
-rwxrwxrwx 1 33333 33333 0 Feb 17 21:28 sevens
---------- 1 33333 33333 0 Feb 17 21:28 zeroes

[aledbf]

@shaal please check #3636

[shaal]

@aledbf Unfortunately, I cannot test it until I get have access to a working build of that PR.
#3636 (comment)