Docker-compose and incorrect absolute paths for volumes

[ej3]

windows 10 build 1503.11 running Xenial 16.04.2 LTS with docker client 17.03.0 connecting to the hyper-V docker-for-windows daemon 17.03.1 and docker-compose 1.11.1

From within wsl things like docker run --rm -v c:/my/stuff:/var/goes/here bash ls /var/goes/here work fantastically. This command will simply list the contents of c:/my/stuff.

What does not work is If you have a docker-compose.yml that looks like:

version: '2'
  services:
    bash_it:
      image: bash
      volumes:
        - ./:/var/goes/here

and you run something like docker-compose -p test -d bash_it up && docker exec test_bash_it_1 ls /var/goes/here. Everything will explode and die - you may have to restart the docker-for-windows daemon / engine.

the error will look something like:

ERROR: for proxy  Cannot start service proxy: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"ro
otfs_linux.go:54: mounting \\\"/mnt/c/my/stuff\\\" to rootfs \\\"/var/lib/docker/overlay2/e4bda146bae2ce38c13ad8f3c4ff5f6e
6f3198041ce02d4c6f593f65948e7086/merged\\\" at \\\"/var/lib/docker/overlay2/e4bda146bae2ce38c13ad8f3c4ff5f6e6f3198041ce02d4c6f593f65948e7086/merged/var/goes/here\\\"
 caused \\\"not a directory\\\"\""
: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
ERROR: Encountered errors while bringing up the project.

It's all rather complicated looking, but I'm pretty sure I can tell you what's going on: the windows-for-docker daemon expects a valid windows path (eg c:/my/stuff) BUT docker-compose is calling os.path.abspath to resolve the absolute path of the resource and it's getting a URI/path that looks like /mnt/c/my/stuff - which is absolute nonsense unless you're in the windows linux subsytem.. which the hyper-v daemon is clearly not.

so... that's an interesting problem. Is there a switch like MSYS_NO_PATHCONV=1 or something I can do to prevent os.path.abspath from returning a wsl alias? OR - how should this work?

[aseering]

So, this is a little more conceptually complicated: Unlile in Cygwin, /mnt/c/... is not an alias in WSL. It is the canonical and only way to refer to that directory. C:\... is not a valid Linux path; therefore not a valid WSL path. (C:\ would immediately cause errors on Linux, so WSL does likewise.)

In this particular case, maybe you could try putting an absolute Windows path in your config file? You might also want to bring this up on the Docker forums -- they seem to be pretty active, and the Docker team seems to care a lot about compatibility. (And this would be a really easy, if kind of hackish, thing for them to work around.)

[sakai135]

os.path.abspath returning /mnt/c/path within WSL makes total sense though. The issue is the mismatch between /mnt/c of WSL and /c in the VM of Docker for Windows.

Within the Hyper-V VM, the C: is mounted to /c instead of /mnt/c. I believe docker/for-win#371 would be the ideal solution, which is to make Docker for Windows mount C: into /mnt/c inside the VM.

[therealkenc]

Within the Hyper-V VM, the C: is mounted to /c instead of /mnt/c

And in MSYS as well, which is how the whole Windows Git BASH world sees things. I know that mounting remote shares on / is an anathema on Real Linux, which WSL strives to follow. But there is (unfortunately) a lot of precedent for that convention, and WSL goes against the grain here. I imagine this will sort itself once WSL supports mounting of DrvFS on arbitrary paths.

[aseering]

@sakai135 -- ah, interesting: I initially assumed Docker for Windows did Windows path translation server-side but it would make a lot of sense for it to do it client-side (in which case the Linux client would indeed need to pass in a VM-compatible path).

@therealkenc -- ooh, that's a good thought: WSL already supports bind-mounts. Therefore it does (in essence) support mounting DrvFS on arbitrary paths. So, proposed workaround (untested):

$ sudo mkdir /c
$ sudo mount --bind /mnt/c /c
$ cd /c/path/to/project
$ docker-compose ...

[sakai135]

@aseering the bind mount worked for me :) Thanks for the workaround!

[therealkenc]

I forgot binding works. Perfect. Once real init (aka systemd) is working, I am imagining that line being in everyone's startup script....

[ej3]

@aseering you made me realize that it was unusual that the WSL docker client was taking non-canonical paths as arguments, so I went back and tried

$ docker run --rm -v /mnt/c/Users/ej3/dev/play/:/var/play bash ls -la /var/play
total 4
drwxr-xr-x    5 root     root           100 Apr  5 21:05 .
drwxr-xr-x    1 root     root          4096 Apr  5 21:47 ..
drwxr-xr-x    2 root     root            40 Apr  5 20:58 Dockerfile
drwxr-xr-x    2 root     root            40 Apr  5 21:02 configure.file
drwxr-xr-x    2 root     root            40 Apr  5 21:05 test.dev.conf

which seemed to work! (but the listing wasn't complete?!) so then I had no idea what the problem could have been, and I panicked a bit.

Then the conversation went on, and the bind for DrvFS came up:

$ docker run --rm -v /c/Users/ej3/dev/play/:/var/play bash ls -la /var/play
total 12
drwxr-xr-x    2 root     root          4096 Apr  5 21:44 .
drwxr-xr-x    1 root     root          4096 Apr  5 21:47 ..
drwxr-xr-x    2 root     root             0 Apr  5 21:44 .git
-rwxr-xr-x    1 root     root            81 Apr  5 21:10 Dockerfile
-rwxr-xr-x    1 root     root           675 Apr  5 21:44 README.md
-rwxr-xr-x    1 root     root           139 Apr  5 21:19 bash_it.sh
-rwxr-xr-x    1 root     root           116 Apr  5 21:04 docker-compose.yml
-rwxr-xr-x    1 root     root            39 Apr  5 21:09 test.conf
-rwxr-xr-x    1 root     root            40 Apr  5 21:08 test.dev.conf
$

Then I was able to reproduce my issue with a simple docker-compose example, and the bind completely resolves the issue.

Just a further note: When I first tried using the bind I was getting a new set of IO errors because I had created the containers with docker-compose on the /mnt/c volume, moved to the /c volume and simply did a docker-compose up without recreating the containers from the images. If anyone else has this problem: be sure to re-instantiate new containers from the original images after you switch mount points.

Thanks.

[teoulas]

I wanted to do the same a few months ago. I ended up simply symlinking /mnt/c to /c:

sudo ln - s /mnt/c /c

Not sure if it's better performance-wise than bind-mounting, but it works just as well.

[aseering]

For what it's worth, there are some applications (Web servers in particular) that will deliberately refuse to read files from any path that contains a symlink. Usually for security reasons, because malicious users could use symlinks to escape DocumentRoot. Bind mounts usually don't suffer from this limitation -- anyone can create a symlink anywhere, but you need elevated permissions (traditionally, root) to set up a bind-mount, so sysadmins can still use them but they're harder to exploit.

[ghost]

Hi folks,

We're talking this over internally at Microsoft and open to suggestions.

From Russ on the WSL (Windows Subsystem for Linux) team: "Bind and symlinks both work, but the bind may be better in some cases since not all services will follow symlinks. Downside of bind is that it will not stick between WSL instances due to our init."

Essentially agreeing with @aseering's points :)

What ideas do you have? What are you trying to do when you run into problems around file paths?

Cheers,
Myles

[tsafin]

From my holistic point of view, the ideal solution would be for all us (and I really expect it happen some day), iff WSL will converge with Docker and Docker on Windows will move to use WSL compatibility layer instead of MobyLinux VM. In this case Docker Compose could be used the way it is used on Linux and transparently rely on WSL underneath.

[ghost]

Thanks for the suggestions!

[edrevo]

Do you need to sudo mount --bind /mnt/c /c everytime you open up a bash? I've tried adding the following line to /etc/fstab but that doesn't seem to do the trick:

C: /c drvfs rw,noatime 0 0

[DeadPixelz01]

Has anyone found a way to make sudo mount --bind /mnt/c /c permanent yet? I know I could just add it to something like ~/.zshrc .However, that requires me to enter a password every time I want to open the terminal >.>. It would be nice if we could add it to something like /etc/rc.local and have it automatically start up when I run the terminal

[therealkenc]

There's a long thread regarding daemons and startup tasks in #834, with a reasonable work-around at the very bottom. Short story, add mount to your /etc/sudoers so you don't have to enter the password every time. The User Voice is here.

[copitz]

@DeadPixelz01 For me docker/for-win#371 (comment) works quite well

[reidblomquist]

oh my god @aseering ily. you just helped me finally jump through the last of the many fiery hoops that line the pathway to getting docker fully operable from within lxss

[pd93]

@aseering Not to pester, but is there any update on this? I'm struggling to get the bind mount workaround to work reliably. IMO, This is one of the last things holding back the WSL finally being a decent developer environment on Windows.

[aseering]

Hm, I don't know of any update. But I'm not actually a WSL dev; just a user. Maybe someone on the WSL team can comment?

For the bind-mount, what have you tried? And have you tried a symlink instead? We might be able to help you with those, too.

[pd93]

@aseering Thanks for the offer of help. I did eventually get this working with the bind mount. Turned out my config was the issue - of course.

Today Microsoft released insiders build 17093. In the release notes it states:

WSL now processes the /etc/fstab file during instance start [GH 2636].

I thought this might be of interest to @edrevo who previously mentioned fstab was not working. Would this help us to keep the bind mount permanent rather than entering it into our .bashrc or equivalent?

The notes also mention:

AF_UNIX allows socket connections between Linux processes on WSL and Windows native processes

Would anyone with a bit more knowledge be able to say if these changes affect this issue at all?

[benhillis]

@pd93 - yes the addition of fstab support adds a better way to configure mounts. AF_UNIX support is new functionality that allows WSL processes and Windows processes to communicate and should not impact this. See the blog post for more details.

[aseering]

Regarding AF_UNIX, in principle Docker for Windows should be able to use that to allow WSL/Linux applications to talk over a local socket (just like native Linux Docker) rather than TCP/loopback. I haven't read this part of Docker's source code, but I would guess that they would need to make some small modifications to their code: Previously, AF_UNIX didn't work and wasn't available on Windows, so existing applications will often avoid even trying it on Windows.

[villasv]

There is OTOH one new feature that might be of relevance here. On wsl.conf you can now specify the mount point of the C: drive:

Key: root	Value: String value (/mnt/…)
Default: “/mnt/”
This key lets you define a new directory where fixed drives will be automatically mounted. For example, if you have a directory in WSL at “/windir/” and you specify that as the root, you would expect to see your fixed drives (for example, C:/) mounted at “/windir/c”

It also added other DrvFS options handling.

[pd93]

@villasv I can confirm that adding the following to the /etc/wsl.conf file on 17093 appears to work:

[automount]
root = /

I brought up my docker project and can access all of my volume mounts 👍

[pd93]

@benhillis Is there a way to obtain the root mount point from from the wsl command line without having to parse the wsl.conf file?

[benhillis]

@pd93 - No there is not.

[aseering]

@pd93 you can get similar information by parsing /proc/mounts. Not clear that's any better :-) But it's more Linux-ey; existing Linux tools already support it.

[dwatrous]

I can get this to work, but it doesn't respect the file permissions I set in WSL. Thanks to recent changes, I can set keys on my windows system to 600 through WSL (I think it uses the drvfs filesystem and attaches some metadata to it). These changes aren't visible on the volume mount. Every file ends up with the same permissions, just like when I run docker from powershell.

[eiva]

How to access WSL filesystem from composer?
Each folder I ‘mount’ using volumes command appear empty in container...
The workarounds described here shows how to access windows drives, not WSL...

[kskalski]

Did anybody experience mounted volumes to list just partial contents when accessed from container? I followed the hints provided here to mount bind /mnt/c as /c and /mnt/d as /d, as well as using symlinks, but the behavior of listing volume from docker container sometimes shows empty dir, sometimes only some of the files/directories that are present on host, sometimes all contents.

I'm using Docker Toolbox with docker machine on Virtual Box and command like
docker run --rm -v /d/rep/:/var/play bash ls -la /var/play
to test

[ej3]

@kskalski I did before mounting the volume to the root. This could be happening because your mount --bind /mnt/d /d is not actually occurring / executing cleanly?

[kskalski]

The mounts seems to be working fine, I can access all content without problem in /d, however when I mount /d as volume to docker container, it shows empty dirs (seems like it memoizes the subdirs that I tried to mount as volumes and shows just them, so running with /d, then /d/subdir, then /d will show subdir as content in last step).

It works somewhat better for /c than for /d though... For /c I can correctly access /c/Users/kamil and all contents in subdir, for /d there is nothing at all. Maybe Docker Toolbox VM is not actually mounting D: and for C: maybe it's a different problem.

[sawyer-peterson]

After fighting with Windows and Docker for a bit, I found out the secret to mapping the drives is as follows:
Update Docker so that it has access to the drive in question (see screenshot), and begin the path using /[drive] (in this case, /c/whatever)

[kskalski]

Ah, ok. For Docker Toolbox (using VirtualBox) you can actually map drives (host C: or D:) under (guest, visible to docker containers) /mnt/c /mnt/d directories, so the workarounds mentioned in this issue could be skipped and the problem solved through VM settings.

[konczak]

Hello!
Yesterday I've had same issue with mounting folders with Docker-compose and this tip has helped me:
docker/compose#4303 (comment)

it was just as simple as "Reset credentials" in Docker "Shared drives", update password to current and restart Docker (in reset menu).

Maybe it will help somebody too.

[repos-jmp]

I faced the same issue with docker-compose as well, by default Docker Toolbox for Windows only has access to C:\Users. Additional directories can be shared via VirtualBox UI.
REF : https://docs.docker.com/toolbox/toolbox_install_windows/#optional-add-shared-directories

[Ciantic]

I think the bind trick is a hack, this should be made to work:

     volumes:
     - "./db:/var/db"

Having to bind the drive first, then navigating to that new fake drive, makes things really complicated. If you already have a development setup with scripts that run the docker-compose, this is annoying thing. I can't go and edit the development scripts because they are correct.

Right now I should do a new script just for WSL, and that is clunky.

[craigloewen-msft]

Please refer to wsl.conf to help with your configuration settings. You may also find this post on using Docker with WSL to be useful