Research: Better bind mounting UX for pack build on Linux

[jromero]

Problem Description

The current experience for bind mounts using pack build -v on Linux is poor due to the lack of isolation/translation that is otherwise provided by Docker Desktop on macOS and Windows. The struggles commonly involve permissions and ownership.

Expected Outcome

It would be great if we could find a possible solution to this problem by simulating what Docker Desktop does.

Questions that we should answer:

1. What does Docker Desktop do? Is this open sourced?
2. What is the reason docker for Linux doesn't have this facade?
3. Are there any community provided solutions that others are using?
4. If we needed to implement this ourselves, what would that look like?

Additional context

Slack conversation transcript:

Ben Hale 3 hours ago
What are you thinking?

David Syer 3 hours ago
I don't know. Is there any precedent for getting reports out of a build?

Ben Hale 3 hours ago
Other than that it’s been a pain since the CI agents started 15 years ago, no. But we’re in a lot of control, so what experience are you looking for? To get some files out (a volume mount might be good enough), keeping the build container around and poking into it, etc?

David Syer 2 hours ago
Volume mount would do it probably

David Syer 2 hours ago
I can dump a load of logs into a file with spring-graalvm-native.

David Syer 2 hours ago
But they never show up on the host obviously

David Syer 2 hours ago
The problem with volume mounts is file permissions, right?

David Syer 2 hours ago
So you could end up creating files in a project that cannot be removed by the user. Build fails. Etc.

Ben Hale 2 hours ago
Docker being docker, it depends on the platform. On macOS, there’s an implicit chmod into and out of the volume mount. On Linux, you’d better hope there are consistent UIDs or you’ll have to sudo chmod yourself.

Ben Hale 2 hours ago
But, as a first try you might actually pack build —volume ${HOME}/native-image-logs:/tmp/native-image/logs ... and see how that goes. I’ll bet it gives you what you want. (edited)

Ben Hale 2 hours ago
If you do decide to experiment, I’d be really interested in your experiences there.

Ben Hale 2 hours ago
(/cc @jromero and @ekcasey another arbitrary volume mount use-case beyond Maven caching)

David Syer 2 hours ago
I'll give that a go. Then if it works I can ask Spring Boot to expose it

[...]

David Syer 1 hour ago
It doesn't quite work

David Syer 1 hour ago
java.io.FileNotFoundException: /tmp/native-image/dump.txt (Read-only file system)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:237)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:187)
at org.springframework.graalvm.support.ReflectionHandler.dump(ReflectionHandler.java:135)
at org.springframework.graalvm.support.SpringFeature.beforeAnalysis(SpringFeature.java:111)

David Syer 1 hour ago
pack seems to be creating the directory as root, and then tries to write to it as non-root:
$ ls -ld samples/petclinic/target/native-image/
drwxr-xr-x 2 root root 4096 Sep 24 16:48 samples/petclinic/target/native-image/

David Syer 1 hour ago
My command line:
$ pack build -e BP_BOOT_NATIVE_IMAGE=1 --volume pwd/target/native-image:/tmp/native-image petclinic:pack

David Syer 1 hour ago
If I jump in and change that directory to writable before it gets to that stage in the build it works. But that's ugly.

Emily Casey 1 hour ago
pack build might be defaulting to readonly volumes. Try --volume $PWD/target/native-image:/tmp/native-image:rw

David Syer 1 hour ago
I did that as well.

David Syer 1 hour ago
It still gets created as root:root so I can't write

[...]

David Syer 1 hour ago
It looks like if the directory already exists it is not modified.

David Syer 1 hour ago
So I can create it world-writable and that probably works.

David Syer 1 hour ago
I'm guessing pack is going to create files in that directory that I can't delete, unless I happen to have the same userid. But at least I see the data.

Emily Casey 1 hour ago
I know the docker --volume flag (on MacOS) generally does some magic to translate users on bind mounts so they are writable by the container user but new files appear to be owned by the host user on the host filesystem. I would guess that either
pack is using the docker client library in a way that doesn’t produce the desired behavior
we are starting the container as root and then dropping privileges and that breaks the user translation
(edited)

Emily Casey 1 hour ago
I can take a quick look

Emily Casey 42 minutes ago
@dsyer which builder image are you using?

Emily Casey 39 minutes ago
Some context on docker bind mount owership - https://docs.docker.com/docker-for-mac/osxfs/#ownership

Docker DocumentationDocker Documentation
File system sharing (osxfs)
osxfs is a new shared file system solution, exclusive to Docker Desktop for Mac. osxfs provides a close-to-native user experience for bind mounting macOS file system trees into Docker containers....

David Syer 37 minutes ago
pack set-default-builder paketobuildpacks/builder:base

Emily Casey 35 minutes ago
2. we are starting the container as root and then dropping privileges and that breaks the user translation
I think we can rule this theory ^ out.
When using docker’s --volume flag the ownership of the bind mounted directory appears always matches the process uid/gid even if we change uid/gid after starting the container
docker run -u root --volume $HOME/.m2:/home/cnb/.m2 -it paketobuildpacks/builder:base bash
root@358525904c3a:/layers# ls -al /home/cnb
total 24
drwxr-xr-x 1 cnb cnb 4096 Sep 24 16:17 .
drwxr-xr-x 1 root root 4096 Sep 17 13:09 ..
-rw-r--r-- 1 cnb cnb 220 Apr 4 2018 .bash_logout
-rw-r--r-- 1 cnb cnb 3771 Apr 4 2018 .bashrc
drwxr-xr-x 4 root root 128 Apr 29 00:37 .m2
-rw-r--r-- 1 cnb cnb 807 Apr 4 2018 .profile
root@358525904c3a:/layers# su cnb
cnb@358525904c3a:/layers$ ls -al /home/cnb
total 24
drwxr-xr-x 1 cnb cnb 4096 Sep 24 16:17 .
drwxr-xr-x 1 root root 4096 Sep 17 13:09 ..
-rw-r--r-- 1 cnb cnb 220 Apr 4 2018 .bash_logout
-rw-r--r-- 1 cnb cnb 3771 Apr 4 2018 .bashrc
drwxr-xr-x 4 cnb cnb 128 Apr 29 00:37 .m2
-rw-r--r-- 1 cnb cnb 807 Apr 4 2018 .profile
(edited)

David Syer 31 minutes ago
Looks a bit different to me
$ docker run -u root --volume $HOME/.m2:/home/cnb/.m2 -it paketobuildpacks/builder:base bash
root@d6644e7911d7:/layers# ls -al /home/cnb
total 28
drwxr-xr-x 1 cnb cnb 4096 Sep 24 16:23 .
drwxr-xr-x 1 root root 4096 Sep 17 13:09 ..
-rw-r--r-- 1 cnb cnb 220 Apr 4 2018 .bash_logout
-rw-r--r-- 1 cnb cnb 3771 Apr 4 2018 .bashrc
drwxrwxr-x 5 cnb cnb 4096 Jun 23 16:59 .m2
-rw-r--r-- 1 cnb cnb 807 Apr 4 2018 .profile

Emily Casey 30 minutes ago
which version of docker desktop?

David Syer 30 minutes ago
None. Linux

David Syer 29 minutes ago
I think it's because my local user is UID=1000 (same as cnb)

David Syer 29 minutes ago
So ${HOME}/.m2 already belongs to "cnb".

Emily Casey 29 minutes ago
Yeah… none of the bind mount ownership magic works on linux

David Syer 28 minutes ago
$ docker run -u root --volume /opt:/home/cnb/opt -it paketobuildpacks/builder:base bash
root@bfe9d3f24c19:/layers# ls -al /home/cnb/
total 28
drwxr-xr-x 1 cnb cnb 4096 Sep 24 16:26 .
drwxr-xr-x 1 root root 4096 Sep 17 13:09 ..
-rw-r--r-- 1 cnb cnb 220 Apr 4 2018 .bash_logout
-rw-r--r-- 1 cnb cnb 3771 Apr 4 2018 .bashrc
-rw-r--r-- 1 cnb cnb 807 Apr 4 2018 .profile
drwxr-xr-x 4 root root 4096 May 13 08:11 opt

David Syer 27 minutes ago
If pack simply created a directory as the current user if it didn't exist that at least would move us in the right direction.

David Syer 27 minutes ago
To be safe it would have to create it world writable (if the user asked for "rw")

Emily Casey 26 minutes ago
As ben mentioned, on linux you have to manage the uid/gids explicitly but if your uid/gid match the cnb user we should be able to make this work

Ben Hale
Docker being docker, it depends on the platform. On macOS, there’s an implicit chmod into and out of the volume mount. On Linux, you’d better hope there are consistent UIDs or you’ll have to sudo chmod yourself.

Emily Casey 26 minutes ago

If pack simply created a directory as the current user if it didn’t exist that at least would move us in the right direction.
On the host?

David Syer 26 minutes ago
Yes

David Syer 26 minutes ago
I think that would be "making it work".

David Syer 25 minutes ago
It's awkward about the uid/gid

David Syer 25 minutes ago
But if the user creates the directory 777 then it seems to work at least

David Syer 24 minutes ago
I guess pack could even notice that the current user is not UID=1000 and do something different

David Syer 24 minutes ago
But that makes an assumption about the builder image?

Emily Casey 23 minutes ago
I worry that creating a world writable directory is a bit unexpected. I would just use the default umask.

Emily Casey 23 minutes ago
You don’t want the container writing files to your host that the host user doesn’t have permission to read.

Emily Casey 22 minutes ago
I think we need a more holistic solution for linux that mimic the docker desktop behavior on mac

David Syer 22 minutes ago
I guess that's the best compromise

David Syer 22 minutes ago
It's going to fail a lot in ways that are hard to predict or understand

Emily Casey 21 minutes ago
At one point we talked about turning bind mounts off entirely on linux, for the time being, anticipating some of these issues

David Syer 20 minutes ago
it's depressing

Javier Romero 7 minutes ago

At one point we talked about turning bind mounts off entirely on linux, for the time being, anticipating some of these issues

Exactly :this:

It’s great to have your feedback @dsyer since we were really missing those data points.

I think we need a more holistic solution for linux that mimic the docker desktop behavior on mac

I’m in favor of this. Although it will likely require a very heavy lift in understanding and finding the appropriate solution that may ultimately depend on host capabilities. We may be in need of a SME.