Docker 2.3.2.0 and later yield permission denied for /var/run/docker.sock when using Docker in Docker

[ericvn]

• I have tried with the latest version of my channel (Stable or Edge)
• I have uploaded Diagnostics
• Diagnostics ID:

Expected behavior

Using Docker in Docker should work the same as in prior versions, like 2.3.1.0

Actual behavior

Docker 2.3.2.0:

docker run -v /var/run/docker.sock:/var/run/docker.sock -u501: -it --rm docker sh
/ $ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json: dial unix /var/run/docker.sock: connect: permission denied

Docker 2.3.1.0:

docker run -v /var/run/docker.sock:/var/run/docker.sock -u501: -it --rm docker sh
/ $ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
6aecfb8cf30a        docker              "docker-entrypoint.s…"   3 seconds ago       Up 2 seconds                            jovial_shirley

Information

• macOS Version: 10.15.5

Diagnostic logs

Docker for Mac: version...

Steps to reproduce the behavior

docker run -v /var/run/docker.sock:/var/run/docker.sock -u501: -it --rm docker sh
/ $ docker ps

[sdake]

Docker 2.3.1.0:

$ docker run -v /var/run/docker.sock:/var/run/docker.sock -u501: -it --rm docker sh
Unable to find image 'docker:latest' locally
...
Digest: sha256:f9a24a777e813e2692920a86530355e3840d0271464fd486bd8708aa018d0ac6
Status: Downloaded newer image for docker:latest
$ cd /var/run
$ ls -ld docker.sock
srw-rw----    1 root     root             0 Jul  9 01:22 docker.sock
$ uid=501 gid=0(root)

vs

/run $ ls -l docker.sock
srwxr-xr-x    1 root     root             0 Jul  9 01:33 docker.sock

Essentially, the edge version of Docker changes the socket permission.

As far as I can tell, it is necessary to run as root on docker for mac when bind mounting the Unix socket.

Cheers,
-steve

[thaJeztah]

This may be related to a recent change where bind-mounting the socket in a container will now bind-mount the proxied socket. Note that the new behavior looks to match permissions on a native Linux install (socket owned by root, and docker group).

You can still access the non-proxied socket at /var/run/docker.sock.raw, and that looks to work;

docker run -it --rm -v /var/run/docker.sock.raw:/var/run/docker.sock -u501 docker sh
$ ls -la /var/run/docker.sock
srw-rw----    1 root     root             0 Jun 27 19:57 /var/run/docker.sock
$ docker ps
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                            NAMES
0f41fbec7289        docker               "docker-entrypoint.s…"   5 seconds ago       Up 4 seconds                                         unruffled_morse

(I'm not on the Docker Desktop team, but recalled seeing a pull request that made that modification)

[sdake]

@thaJeztah thanks for the response. I don't watch PRs going into docker for mac and appreciate you taking the time to respond. You had mentioned that the proxied docker socket should match the permissions of native Linux (that is the sock file would be owned by GID docker). This does not appear to be the case.

Thanks for the workaround. We will have to sort out how to handle this as our community doesn't run edge consistently (e.g. I run stable).

Cheers,
-steve

[ericvn]

@thaJeztah I'll try the work-around later today (have to reinstall the latest version again) to verify if that might work. It looks like we might be able to check for the existance of the docker.sock.raw and use that instead.

[thaJeztah]

I don't watch PRs going into docker for mac and appreciate you taking the time to respond.

These were in the private repository, so you wouldn't be able to 🤗

You had mentioned that the proxied docker socket should match the permissions of native Linux (that is the sock file would be owned by GID docker)

Sorry, looks like I mixed up my results (switching between a Linux and a MacOS machine 😅)

It looks like we might be able to check for the existance of the docker.sock.raw and use that instead.

Consider it a workaround for now (I think defaulting to use the "proxied" socket was done for a reason; some API calls may need some modifications to account for the way docker desktop is set-up, so skipping that could affect functionality 😅 )

Chatting with @djs55 on our Slack, and he's looking into the issue; thanks for reporting!

[ericvn]

Actually, I quickly installed the 2.3.3.0 version (later than release pointed to by another issue I wrote), and the work-around does indeed work there.

So not sure if this will be considered a regression and fixed or if what's there will continue forward for the foreseeable future.

We can check for the presence the file and use it instead (at least for now).

[sdake]

Thanks. I don't have any reasonable suggestions for how to handle permissions on mac. I was never able to get permission mapping to work well on Linux, so mapping it on Mac is even more of a challenge.

In Istio's docker in docker environment which is used for build and testing and release, it is necessary to run as the normal user's UID in order to write files out of the container in a different host mount (the output of a compile).

I did really give it my all to compile everything in a container and keep the binaries in the container, however, the community found the model of running a container to access binaries/debug tools to be undesirable.

I am hopeful someone else can pick up the work now that a super strong foundation is built. :)

Cheers,
-steve

[ericvn]

@thaJeztah Just curious if there is a way that I can easily check from the command line if I am on one of the DockerForMac versions that require the /var/run/docker.sock.raw (all recent edge versions do). I believe that file did exist in earlier edge releases, but on edge 2.3.6.0, I don't see the .raw anymore:

ls -la /var/run/docker*
lrwxr-xr-x  1 root  daemon    71B Aug 31 08:20 /var/run/docker-cli.sock -> /Users/ericvn/Library/Containers/com.docker.docker/Data/docker-cli.sock
lrwxr-xr-x  1 root  daemon    67B Aug 31 08:20 /var/run/docker.sock -> /Users/ericvn/Library/Containers/com.docker.docker/Data/docker.sock

[ktvoelker]

I've just run into this after upgrading to 2.4.0.0 - I haven't been using edge releases.

I tried the raw socket workaround described earlier, but it did not work (same behavior as without the workaround). I should note that /var/run/docker.sock.raw doesn't exist for me, but I found a socket at /Users/karl/Library/Containers/com.docker.docker/Data/docker.raw.sock which I figured might be the right one, so I tried mounting that in.

[ericvn]

This is my Docker mount point: -v /var/run/docker.sock.raw:/var/run/docker.sock. Note that

ls /var/run/docker.sock.raw
ls: /var/run/docker.sock.raw: No such file or directory

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -u501 docker sh                                                                           
/ $ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json: dial unix /var/run/docker.sock: connect: permission denied
/ $ exit

docker run -it --rm -v /var/run/docker.sock.raw:/var/run/docker.sock -u501 docker sh                                                                     
/ $ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
a3dfce45ce81        docker              "docker-entrypoint.s…"   2 seconds ago       Up 2 seconds                            stupefied_hoover
/ $

Latest Edge version

[grozan]

so same here, running 2.4.0.0

$ docker run -v /var/run/docker.sock:/var/run/docker.sock -u501: -it --rm docker docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json: dial unix /var/run/docker.sock: connect: permission denied

like @ktvoelker, mounting the only docker.raw.sock present on my laptop does not work
Waiting for a real fix to that issue, I found a little workaround which is working for me (until next time I restart docker desktop at least)

docker run -v /var/run/docker.sock:/var/run/docker.sock -u root alpine chmod g+w /var/run/docker.sock

After I "fix" the permissions once, all other containers can access it like before.

$ docker run -v /var/run/docker.sock:/var/run/docker.sock -u501: -it --rm docker docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED                  STATUS                  PORTS               NAMES
d17cabaea882        docker              "docker-entrypoint.s…"   Less than a second ago   Up Less than a second                       intelligent_hugle

so at least I can continue to work on what I need to do.
Restarting docker desktop resets those permissions though

[ericvn]

@grozan, did you try using the non-existent docker.sock.raw: docker run -it --rm -v /var/run/docker.sock.raw:/var/run/docker.sock -u501 docker ps? As noted above, it doesn't exist locally, but when specified does solve my problem on edge.

[priyendra]

Is there documentation on what is this magic non-existent "docker.sock.raw" file?

[grozan]

@ericvn ah no, I had missed that. Mounting something that does not exist... I had not tried.
I can confirm it's working for me too, though, magically enough

[lucymhdavies]

That magic /var/run/docker.sock.raw presumably exists within the Docker for Mac VM, rather than on the macOS host?

(hence why things like -v /var/run/docker.sock.raw:/var/run/docker.sock work)

As others have stated, this feels a bit "magic", and it would be good to either have this magic file documented as the official solution, or to restore group access to the /var/run/docker.sock.raw file (unless there's a good reason why this was changed).

[ktvoelker]

Thanks @ericvn for explaining the workaround. It is working for me.

I hope that the bug can be fixed, but of course I have no idea what that would take.

[ericvn]

Agree that it good so we could determine with mount to use (old vs new) as we have users on older DockerForMac versions and newer ones so we can do this automatically. At one time, I thought the .raw file existed so I could test for it's existence to determine which mount to use. I don't know how to do that now.

[ericvn]

It looks like there is a permission difference on the socket between the edge and stable versions. The stable versions seem to have moved to the new socket, so I need to update the mount point, but then I get another permission error:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/json: dial unix /var/run/docker.sock: connect: permission denied

[matschaffer]

Not sure if this expected but I needed to create that raw symlink myself:

sudo ln -s "$HOME/Library/Containers/com.docker.docker/Data/docker.raw.sock" /var/run/docker.sock.raw

Then mount that to the container:

❯ docker run -v /var/run/docker.sock.raw:/var/run/docker.sock -u501: -it --rm docker sh
/ $ docker ps
CONTAINER ID        IMAGE                                                              COMMAND                  CREATED                  STATUS                          PORTS                              NAMES

This is on docker for mac 2.5.0.1 (49550) - stable.

[ericvn]

@matschaffer I'm not sure if things are hanging around from a previous edge install, but I did try the latest stable 2.5.0.1 yesterday and things worked for me using docker run -it --rm -v /var/run/docker.sock.raw:/var/run/docker.sock -u501 docker ps without creating the symlink. I'm not seeing that my install requires sudo and waiting on others who had required a sudo in the past now work without it.

[matschaffer]

Makes sense. This is fairly new macbook that has only ever had stable installed. My /var/run is drwxrwxr-x 43 root daemon 1376 Nov 12 02:01 . and my user isn't in the daemon group, hence the sudo for my case.

/var/run/docker.sock.raw didn't exist until I created it, but that was pretty easy to see once I saw where docker.sock was linking to.

[ericvn]

/var/run/docker.sock.raw doesn't need to exist to be used as a mount (at least it doesn't exist any of our machines. In my case /var/run is in the wheel group. Not sure on why that is.

[thaJeztah]

Correct; When you bind-mount /var/run/docker.sock.raw in a container, it is mounted from within the VM (so daemon side), so the file doesn't have to exist on the macOS machine.

/var/run/docker.sock on the macOS machine is used by the docker cli (which runs on the mac) to communicate with the daemon inside the VM (the socket on macOS forwards the connection to the daemon inside the VM

(I realize this may be confusing; hope my explanation helps!)

[matschaffer]

Ahhh, thanks for the clarification.

[leighmcculloch]

I think the issue is that the docker.sock within the VM has the wrong permissions and group by default. This is what ls shows me about the docker.sock:

$ ls -lah
srwxr-xr-x 1 root root 0 Nov 17 14:43 /var/run/docker.sock

This is my work around for this, which I run inside my container, but it would be better if this was fixed in the VM within docker-for-mac.

$ sudo chgrp docker /var/run/docker.sock
$ sudo chmod g+w /var/run/docker.sock

Ref: #5072 (comment)

After that work around it becomes:

srwxrwxr-x 1 root docker 0 Nov 17 14:43 /var/run/docker.sock

[doctorpangloss]

I am permanently on 2.3.0.0 because of this issue. I'm not sure what has changed, but I see

panic: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

in later versions, for no good reason

[matschaffer]

I upgraded to 3.0.1 over the weekend and it seems to be working as expected without overriding my docker socket info. Still a little fuzzy on the details of the issue though, but thought it was worth mentioning for people to try.

[ericvn]

Using 3.0.1, the old behavior still fails:

docker run -v /var/run/docker.sock:/var/run/docker.sock -u501: -it --rm docker docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json: dial unix /var/run/docker.sock: connect: permission denied

Changing to docker.sock.raw (workaround noted above) still works:

 docker run -v /var/run/docker.sock.raw:/var/run/docker.sock -u501: -it --rm docker docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                  PORTS               NAMES
b9b9342a484a        docker              "docker-entrypoint.s…"   1 second ago        Up Less than a second                       wonderful_tu

The .raw only exists in the container:

> ls -la /Users/ericvn/Library/Containers/com.docker.docker/Data/docker.sock*
srwxr-xr-x  1 ericvn  staff     0B Dec 14 08:10 /Users/ericvn/Library/Containers/com.docker.docker/Data/docker.sock
> ls -la /var/run/docker.sock*
lrwxr-xr-x  1 root  daemon    67B Dec 11 15:32 /var/run/docker.sock -> /Users/ericvn/Library/Containers/com.docker.docker/Data/docker.sock

[doctorpangloss]

I don't know, every external tool has essentially been broken since 2.3.2.0. Maybe change the title to "/var/run/docker.sock stopped working".

[ericvn]

@DmytroSokhach I did ask earlier if there was a way to tell if I should be using the .raw, and no response so far. I did look and it seems like the .raw exists in the ~/Library/Containers/com.docker.docker/data directory for older versions so testing for its presence isn't enough.

[ericvn]

Still fails on Docker Desktop 3.3.0. Still haven't found a way to determine which mount point to use so things work across all releases.

[DmytroSokhach]

With Docker 3.3.3 adding ".raw" workaround still works for me:
/var/run/docker.sock.raw:/var/run/docker.sock
Any ideas how to workaround it or if any fix expected?

[doctorpangloss]

On Big Sur and latest docker I stopped experiencing the issue.

[ericvn]

I am running MacOS 11.3.1 on Intel with Docker Desktop for Mac 3.3.3, and I have seen no change since I opened the issue:

> docker run -v /var/run/docker.sock:/var/run/docker.sock -u501: -it --rm docker docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json: dial unix /var/run/docker.sock: connect: permission denied
> docker run -v /var/run/docker.sock.raw:/var/run/docker.sock -u501: -it --rm docker docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS                  PORTS     NAMES
3c5b2b06f5ed   docker                          "docker-entrypoint.s…"   2 seconds ago    Up Less than a second             admiring_hawking
7b3a28929c52   moby/buildkit:buildx-stable-1   "buildkitd --allow-i…"   14 minutes ago   Up 14 minutes                     buildx_buildkit_build

[DmytroSokhach]

Just found couple cool sites:

1. Uses "socat" proxy: https://mingheng.medium.com/solving-permission-denied-while-trying-to-connect-to-docker-daemon-socket-from-container-in-mac-os-600c457f1276
2. https://forums.docker.com/t/permission-denied-while-trying-to-connect-to-the-docker-daemon-socket/72570/3
   which has interesting reply:

https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option

Daemon socket option

The Docker daemon can listen for Docker Engine API 14 requests via three different types of Socket: unix , tcp , and fd .

By default, a unix domain socket (or IPC socket) is created at /var/run/docker.sock , requiring either root permission, or docker group membership.

So just read it and now thinking: maybe mapping unix:///var/run/docker.sock inside container is not best option at all, like not "best practice". Maybe better practice is to connect via tcp? (which obviously requires more significant changes).

[leighmcculloch]

A couple months ago I updated Docker on Mac and the fixes above (#4755 (comment)) stopped working for me. I've found the most reliable approach is to use socat to serve the docker /var/run/docker.sock on a TCP port which containers can then access. I use this script: dockersocat, so that it is convenient.

Maybe better practice is to connect via tcp?

I think resorting to TCP is a tradeoff. The Docker daemon already supports TCP, but it is turned off by default: https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option. Socating to TCP, or running Docker with TCP open, is a bit of a security concern. There's no auth, so any application, or any other contain can access the port.

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[thaJeztah]

/lifecycle frozen

[Looooopy]

A couple months ago I updated Docker on Mac and the fixes above (#4755 (comment)) stopped working for me. I've found the most reliable approach is to use socat to serve the docker /var/run/docker.sock on a TCP port which containers can then access. I use this script: dockersocat, so that it is convenient.

Maybe better practice is to connect via tcp?

I think resorting to TCP is a tradeoff. The Docker daemon already supports TCP, but it is turned off by default: https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option. Socating to TCP, or running Docker with TCP open, is a bit of a security concern. There's no auth, so any application, or any other contain can access the port.

I suppose you could have used unix socket instead of TCPIP which is faster and isolate it to be on the same system.

Run this as root in docker then switch to your user

export DEV_USER=not_root
export DEV_UID=1000
export DEV_GID=1000

    socat UNIX-LISTEN:/home/"${DEV_USER}"/.docker/docker_socket,fork,user="${DEV_UID}",group="${DEV_GID}",mode=777 \
      UNIX-CONNECT:/var/run/docker.sock \
      &

    export DOCKER_HOST=unix:///home/${DEV_USER}/.docker/docker_socket

services:
  my-service:
    ...
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ...

[DmytroSokhach]

Solution for docker swarm mode (Docker 4.6.0, MacOS BigSur 11.6.4):

service_that_controls_docker_on_host:
  environment:
    # SOCAT to access Docker service via SOCK
    - DOCKER_HOST=tcp://mystack_socat:2375
  networks:
    - overlay_network

socat:
  image: alpine/socat
  command: TCP4-LISTEN:2375,fork,reuseaddr UNIX-CONNECT:/var/run/docker.sock
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  networks:
    - overlay_network

"mystack" is a name for swarm stack, as well as prefix for service names.
"overlay_network" is optional and only needed in case other services in the swarm are using it.

[prineshaz]

Docker Desktop on MacOS 12.5.1 (Monterey) @thaJeztah workaround worked for me:

/var/run/docker.sock.raw:/var/run/docker.sock on container run command. Thank you!

[ericvn]

It looks like the problem reappeared on Docker 4.19.0. The workaround does not solve the issue this time.

[ericvn]

Fixed in 4.21.1.