Permission denied when accessing .docker-root/overlay2

[sagor999]

Bug description

I see a lot (40k in last 3 hours) errors like this in SaaS:

jsonPayload: {
@type: "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent"
error: "permission denied"
gid: 33333
level: "warning"
message: "cannot chown"
path: ".docker-root/overlay2/205df79f676c4bbc3e8af627dc2a78228e55f494d7b21c5725fcaeb808e285cb/diff/home/node/parse-server/"
uid: 33333

Link to logs: https://cloudlogging.app.goo.gl/mZg3N1J2w58wm7fLA

This might be related to an issue customer is having as well:
https://app.frontapp.com/inboxes/teammates/9932874/inbox/open/41825492426

Potentially related issue: #9263

Would like to understand why we are getting those errors and what is the impact.

Steps to reproduce

One potential repro case:
This seems to do with running docker-compose pull and docker-compose build in init task.

Workspace affected

tracthq-tractweb-xtjbxw26gqh

Expected behavior

We shouldn't be getting permission denied errors when accessing .docker-root I think.

Example repository

No response

Anything else?

No response

[sagor999]

@csweichel would you have any insight as to why we might be getting this error?

[utam0k]

I guess this problem is related to seccomp notify.

gitpod/components/workspacekit/pkg/seccomp/notify.go

Lines 426 to 451 in e24461c

	func (h *InWorkspaceHandler) Chown(req *libseccomp.ScmpNotifReq) (val uint64, errno int32, flags uint32) {
	log := log.WithFields(map[string]interface{}{
	"syscall": "bind",
	"pid": req.Pid,
	"id": req.ID,
	})
	memFile, err := readarg.OpenMem(req.Pid)
	if err != nil {
	log.WithError(err).Error("cannot open mem")
	return
	}
	defer memFile.Close()
	pth, err := readarg.ReadString(memFile, int64(req.Data.Args[0]))
	if err != nil {
	log.WithError(err).Error("cannot open mem")
	return
	}
	if strings.HasPrefix(pth, "/dev/pts") {
	return 0, 0, 0
	}
	return 0, 0, libseccomp.NotifRespFlagContinue
	}

[kylos101]

@utam0k after looking at #9247, maybe consider this issue?

[sagor999]

From another customer via Front:

gitpod.yml is as following:
------------------
tasks:
- init: |
sed -i 's/[email protected]:celtra/https:\/\/github.com\/celtra/g' .gitmodules
git submodule init
git submodule sync
git submodule update --recursive

git clone https://github.com/celtra/uab-devcontainers.git .devcontainer

touch .env .test-env
time docker-compose build
------------------

Prebuild environment successfully builds, but when trying to create a new workspace with it, the following error is shown:

------------------
Oh, no! Something went wrong!
cannot initialize workspace: cannot initialize workspace: prebuild initializer: prebuild initializer: openfdat /dst/.docker-root/overlay2/1htqhv8btix39teqczhlqgcbp/work/work: permission denied
------------------

[utam0k]

This matter has appeared since May 5 when the gen44 was shipped.
https://cloudlogging.app.goo.gl/xCD6MPnXVcY55Fxj6

[utam0k]

Possibly related:

• [docker-up] Update docker compose to 2.5.0 #9699
• [content-service] Fix backup restore #9732

[utam0k]

It appears that I do not have enough permissions to set the attribute user.overlay.impure.
https://cloudlogging.app.goo.gl/dQBLSUSoQboBXVPn9

gitpod/components/content-service/pkg/archive/tar.go

Line 192 in 8c50daf

log.WithField("name", key).WithField("value", value).WithField("file", name).WithError(err).Error("restoring extended attributes")

[kylos101]

@utam0k can you share a repo and test results for starting a regular workspace from a prebuild, where there are both before and init tasks defined, both of which depend on docker compose and files/content that get modified in /workspace?

I was talking with @aledbf , and he shared some interesting questions.

1. When do we wait to start a backup at the end of prebuild, after the before and init tasks have run? Are we waiting to shut down dockerd and related containers, after waiting for before and init tasks to finish, before we start the backup of /workspace?
2. How do we wait to finish content restore for a regular workspace (when restoring from the prebuild), before we start the before task for the regular workspace? Do we need to wait to finish content restore before starting dockerd?
3. We have similar questions for supervisor and related workspace processes unrelated to docker. Are we waiting for supervisor to stop before starting the backup for prebuilds? Are we waiting to start supervisor until content restore is finished?

[kylos101]

Hey @utam0k , also, I am curious, what is the impact if user.overlay.impure attribute cannot be set? I'm not sure if there is user impact, or if it makes troubleshooting harder, etc.

[utam0k]

Hi, @kylos101 and @aledbf. Frankly, I just figured out the cause and have no idea what the implementation will be. However, the points you raised are quite important to consider. And it seems difficult.

This is an excellent point. I too have questions about this, but haven't fully explored the details yet. Perhaps it may be best to ignore it and update the user documentation.

Hey @utam0k , also, I am curious, what is the impact if user.overlay.impure attribute cannot be set? I'm not sure if there is user impact, or if it makes troubleshooting harder, etc.

[utam0k]

And if users have no complaints about the current behavior, it may be safe to ignore it.

[atduarte]

We received one more report of this issue: https://app.frontapp.com/inboxes/teams/views/142058954/open/42064138762

It seems the issue also occurs when restarting workspaces after executing certain commands in the workspace.

[aledbf]

@utam0k we should wait for dockerd to be stopped before starting the backup. Please check if dockerd is properly terminated on shutdown.

[aledbf]

@utam0k here is an explanation for user.overlay.impure https://github.com/hisilicon/overlayfs-progs/blob/e10ef686570d9c7eff42f52461593a5c15da56bd/README#L62

[atduarte]

In case it's helpful, here's another one: https://discord.com/channels/816244985187008514/909791731619344494/974263437767110676

[michielbdejong]

I'm still seeing this error:

cannot initialize workspace: cannot initialize workspace:
prebuild initializer: prebuild initializer:
openfdat /dst/.docker-root/overlay2/04ea7...98cb/work/work:
permission denied

This is with the prebuild from muze-nl/federated-timesheets-tests#2

[kylos101]

Hey @michielbdejong , we're hoping to resolve in #11183, which now has a concrete set of steps for us to reproduce the problem, as well as prevent other related regressions. Please subscribe to #11183 for updates?