Error connecting to Docker daemon on macOS with Docker Desktop 4.13

[scottfrederick]

When using Docker Desktop 4.13, the Maven spring-boot:build-image goal and Gradle bootBuildImage task might fail with an error message like this:

Connection to the Docker daemon at 'localhost' failed with error "[2] No such file or directory"; ensure the Docker daemon is running and accessible

via @trisberg and @danvega

[scottfrederick]

On Linux, Spring Boot tries to talk to the Docker daemon using a socket at /var/run/docker.sock. On macOS with Docker Desktop, this has apparently been a symlink to $HOME/.docker/run/docker.sock. Starting with Docker Desktop 4.13.0, this symlink is no longer created.

For Mac
* By default Docker will not create the /var/run/docker.sock symlink on the host and use the docker-desktop CLI context instead.

The "docker-desktop CLI context" refers to the context that can be seen with docker context ls. Contexts listed there map to files in $HOME/.docker/contexts/meta/[sha]/meta.json, except the default context. The endpoint for the default context is determined internally by the docker CLI.

It might be nice for the Spring Boot plugins to inspect $HOME/.docker/config.json to see if a context is specified there, and use the endpoint from the selected context. That won't help for this default context situation though, where there is no context specified in config.json and no meta.json file containing the endpoint.

I think we'll need to default to $HOME/.docker/run/docker.sock on macOS and /var/run/docker.sock for other Linux variants to address this.

[wilkinsona]

I think we'll need to default to $HOME/.docker/run/docker.sock on macOS and /var/run/docker.sock for other Linux variants to address this

That sounds good to me.

[philwebb]

They're going to roll the change back to give folks more time to adapt: docker/for-mac#6529 (comment)

[scottfrederick]

Marking this as blocked for now. There are work-arounds available, and once Docker can figure out a more orderly upgrade path with docs for client library authors there might be a better way to detect the best socket to use on all platforms.

[danshome]

@philwebb Is there any workaround for this? This is blocking all of our development on MacOS/IntelliJ and Docker Desktop 4.14.0 (91374). I just noticed a new build and am installing it now, but as of 4.14.0 (91374) not only is $HOME/.docker/run/docker.sock not getting created any longer, but neither is /var/run/docker.sock.

[danshome]

@philwebb Looks like the latest build fixes it, now unix:///Users/USER/.docker/run/docker.sock is getting created again, but a new issue has shown up. Looks like -f was removed from the docker command so now IntelliJ doesn't work anymore. Nice. It gets the error "unknown shorthand flag: 'f' in -f" changing IntelliJ not to use Compose V2 works, but not ideal.

[morrica]

Docker Desktop is now the default install on Linux as well as Mac. See https://docs.docker.com/desktop/install/linux-install

The differences in socket location and context now cause problems for users of Linux (and probably Windows) as well. Given that most users will be moving to Docker Desktop installations and probably using that UI for development, perhaps it would be a good idea to make an effort to improve build-image usability with Docker Desktop out of the box, without workarounds.

[igloo12]

I had to enable the daemon on docker desktop see

https://stackoverflow.com/questions/74173489/docker-socket-is-not-found-while-using-intellij-idea-and-docker-desktop-on-macos

Also
id("org.springframework.boot") version "3.1.1"
didn't work

id("org.springframework.boot") version "3.0.6"
did

[scottfrederick]

@morrica What's the problem you're having with Docker Desktop on Linux? Is Docker Desktop creating a socket in a location other than /var/run/docker.sock, or not creating a socket at all?

[morrica]

There are three issues I have run into attempting to use build-image with Docker Desktop.

1: I initially received a "permission denied" error when running mvn spring-boot:build-image.
I believe this is because on install Docker Desktop installs the docker service at the standard socket but does not add the user to the docker group. I could be wrong about this and it could be a quirk of my system as I had recently done a system upgrade, removed the previous docker install, and installed Docker Desktop. In any case, adding myself to the group solved that error and build-image appeared to work.

2. At this point the build image was nowhere to be found.
The build-image task reported the image built and listed name of it at the end of the output, but the image did not appear in the Docker Desktop UI. In addition, it was not found when running docker image ls from the command line.

It took some searching to figure out that the UI and docker command were both using a context that pointed to $HOME/.docker/desktop/docker.sock. docker context ls showed the following:

NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT                                    KUBERNETES ENDPOINT   ORCHESTRATOR
default             moby                Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                                              
desktop-linux *     moby                Docker Desktop                            unix:///home/morrica/.docker/desktop/docker.sock

This was solved by adding the correct host to my project's pom.xml:

<configuration>
    <docker>
        <host>unix:///home/morrica/.docker/desktop/docker.sock</host>
    </docker>
    ...
</configuration>

3. Finally there is an issue running an image created by build-image from the Docker Desktop UI.
The Desktop UI does not allow you expose a port that was not specified with an EXPOSE line in the Dockerfile. The built images can be run with docker run -p 8080:8080 myorg/myimage and then the resulting containers can be manipulated through the Desktop UI, but an image initially run through the UI will not have any exposed ports, so is mostly useless. I believe exposed ports will need to be added to the available build-image config for it to be useful with the Desktop UI.

Clearly only issue 2 above relates directly to this ticket, but I provide them all here so that you have a full picture of the first run experience for a new user. Hopefully changes can be made that will return build-image to a "just works" status with Docker Desktop.

Thank you @scottfrederick for looking into it :-)

[scottfrederick]

Thanks for the detail @morrica.

Clearly only issue 2 above relates directly to this ticket

That is correct. This is made even worse by the fact that running Docker Engine and Docker Desktop together is a supported configuration which causes confusion when Spring Boot uses the default endpoint unix:///var/run/docker.sock always but the docker CLI will use its current context. In your case, the image was built successfully but in the "wrong" daemon from your perspective.

To address this I think we'll have to make the changes I suggested above. Our concern with this approach has been that the Docker config.json file and the meta.json files that the CLI uses to configure context aren't specified and could change without notice.

[morrica]

@scottfrederick Perhaps calling the docker context ls or docker context inspect CLI commands directly would be an option. The Docker Desktop install adds them so they should be available in either case. Not sure if the output would be more stable than json config files you mentioned though.

[ThomasVitale]

When installing Docker Desktop on macOS, there is a setting to create both /var/run/docker.sock and /Users/<user>/.docker/run/docker.sock, with the first one being a link to the second one (the actual socket). That setting should make it so that the many tools relying on the standard Docker socket (/var/run/docker.sock) continue working without additional configuration. For the time being, it might be an idea to add that information to the Spring Boot documentation for Buildpacks.

Regarding defaulting Spring Boot to /Users/<user>/.docker/run/docker.sock for macOS, that would currently break the compatibility with Podman which is also affected by the socket change and is currently looking into a solution to support both in a transparent way (see issues here and here). It would be nice if there was a way for Spring Boot to use the current context automatically, whatever that is.

[scottfrederick]

I've created #36445 to start using the Docker CLI context to determine the daemon host address to use. We'll leave this issue open in case we need to address the /var/run/docker.sock vs $HOME/.docker/run/docker.sock default socket value when the default context is used.

[philwebb]

Given that Docker have restored the old behavior we'll close this one for now. If they break things again in the future we can reopen it.