Docker in Docker (dind) container linking problem on macOS

[Uplink03]

Expected behavior

docker info should just work:

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
...

Actual behavior

docker info cannot resolve the hostname docker, but it works fine it pointed directly to the IP address:

$ docker info
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
$ export DOCKER_HOST='tcp://172.17.0.2:2375'
$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
...

Information

Docker for Mac: version: 1.13.1 (94675c5a7)
macOS: version 10.12.3 (build: 16D32)
logs: /tmp/70A59DC6-BF23-4515-BC34-03CC59FF47A3/20170214-181939.tar.gz
[OK] vmnetd
[OK] dns
[OK] driver.amd64-linux
[OK] virtualization VT-X
[OK] app
[OK] moby
[OK] system
[OK] moby-syslog
[OK] db
[OK] env
[OK] virtualization kern.hv_support
[OK] slirp
[OK] osxfs
[OK] moby-console
[OK] logs
[OK] docker-cli
[OK] menubar
[OK] disk

Steps to reproduce the behavior

In the macOS Terminal:

docker run -d --name docker --privileged docker:1.13-dind
docker run -ti --rm  --name client --link docker:docker docker:1.13 /bin/sh

In the second container's shell:

$ docker info
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

$ ping -c 1 docker
PING docker (172.17.0.2): 56 data bytes
64 bytes from 172.17.0.2: seq=0 ttl=64 time=0.120 ms

$ export DOCKER_HOST='tcp://172.17.0.2:2375'
$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.8-moby
Operating System: Alpine Linux v3.5 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952 GiB
Name: 04032b297320
ID: 6SVY:P6MV:EIGT:PCWX:KXLL:2VX6:C62Y:ERUL:3AJL:3X3T:KXPV:SKEZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

The cause - but only on Mac?

In the second container:

$ cat /etc/resolv.conf
# Generated by dhcpcd from eth0.dhcp
# /etc/resolv.conf.head can replace this line
nameserver 192.168.65.1
# /etc/resolv.conf.tail can replace this line

$ ping -c 1 192.168.65.1
PING 192.168.65.1 (192.168.65.1): 56 data bytes
64 bytes from 192.168.65.1: seq=0 ttl=37 time=0.630 ms

$ strace -f docker info
...
connect(3, {sa_family=AF_INET, sin_port=htons(2375), sin_addr=inet_addr("81.200.64.50")}, 16) = -1 EINPROGRESS (Operation in progress)
...

$ cat /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.17.0.2	docker 04032b297320
172.17.0.3	ad3d5606774c

Where does 192.168.65.1 come from? I seem able to ping many addresses that are unpingable from the host (4.4.4.4, 254.254.254.254, 1.3.3.7, ...)

Why is it resolving 'docker' to '81.200.64.50'?

Going to that IP I get a Virgin Media page. I'm with Virgin Media coincidentally. It says something about 'Advanced Network Error Search', but I've never seen that ever before.

In the corner it says: This service is optional and you can switch it off here

So I clicked on it:

WTF IS THAT?

I switch to 'No' and... HURRAH! Magically and immediately the problem is gone. I don't even have to restart any containers...

Why do I get this only on my Macs? It doesn't happen in Linux.

[ijc]

@Uplink03 thanks for your report.

Going to that IP I get a Virgin Media page. I'm with Virgin Media coincidentally. It says something about 'Advanced Network Error Search', but I've never seen that ever before.

This is an obnoxious behaviour of the Virgin Media (who I am also with at home) DNS resolver which redirects NXDOMAIN responses to DNS lookups with a redirection to an opportunity for them to advertise at you. I would strongly recommend turning it off (and maybe complain to them while you are at it). Perhaps your Linux machines are not using Virgin's DNS resolvers (e.g. using 8.8.8.8 instead)?

The upshot is that your lookup of docker is failing and you are getting sent to their portal.

The question though is why this is going to your upstream nameservers at all (since they clearly won't know about your containers) rather than being handled locally by docker. I tried your repro in our office (which isn't on Virgin) and things just worked with no messing.

stracing I see:

/ # strace -e stat,openat,connect -f docker info
strace: Process 77 attached
strace: Process 78 attached
strace: Process 79 attached
[pid    76] openat(AT_FDCWD, "/proc/sys/net/core/somaxconn", O_RDONLY|O_CLOEXEC) = 3
[pid    76] stat("/root/.docker/config.json", 0xc4203d25e8) = -1 ENOENT (No such file or directory)
[pid    76] stat("/root/.dockercfg", 0xc4203d26b8) = -1 ENOENT (No such file or directory)
[pid    76] stat("/usr/local/sbin/docker-credential-secretservice", 0xc4203d2788) = -1 ENOENT (No such file or directory)
[pid    76] stat("/usr/local/bin/docker-credential-secretservice", 0xc4203d2858) = -1 ENOENT (No such file or directory)
[pid    76] stat("/usr/sbin/docker-credential-secretservice", 0xc4203d2928) = -1 ENOENT (No such file or directory)
[pid    76] stat("/usr/bin/docker-credential-secretservice", 0xc4203d29f8) = -1 ENOENT (No such file or directory)
[pid    76] stat("/sbin/docker-credential-secretservice", 0xc4203d2ac8) = -1 ENOENT (No such file or directory)
[pid    76] stat("/bin/docker-credential-secretservice", 0xc4203d2b98) = -1 ENOENT (No such file or directory)
[pid    76] openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid    76] openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
[pid    76] stat("/etc/mdns.allow", 0xc4203d2e08) = -1 ENOENT (No such file or directory)
[pid    76] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.65.1")}, 16) = 0
strace: Process 80 attached
[pid    76] connect(5, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.65.1")}, 16) = 0
[pid    80] connect(5, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.65.1")}, 16) = 0
[pid    80] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.65.1")}, 16) = 0
[pid    78] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.65.1")}, 16 <unfinished ...>
[pid    80] connect(5, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.65.1")}, 16) = 0
[pid    78] <... connect resumed> )     = 0
strace: Process 81 attached
[pid    81] stat("/etc/hosts", {st_mode=S_IFREG|0644, st_size=205, ...}) = 0
[pid    81] openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3

So it does indeed appear to be asking the network before consulting /etc/hosts which seems pretty odd to me but I don't think that is a docker for mac issue. The client here is the statically linked docker client CLI and /etc/hosts is generated by the host docker engine so please would you file an issue against https://github.com/docker/docker to investigate what is going on (it could be a Go issue, but docker/docker is the right place to go next I feel).

I'm going to close this one for now, but please let us know if this does somehow turnout to be docker ofr mac specific.

[Uplink03]

I haven't opened the issue with the other repository as recommended (I didn't have time to think how to present it), but I'm putting this note here with a workaround for people who need a fix immediately.

Force Google DNS in Docker for Mac using a custom config:

{
  "dns" : [
    "8.8.8.8",
    "8.8.4.4"
  ]
}

This goes in: Preferences -> Daemon -> Advanced.

I had npm install trigger some weird DNS and networking issues in containers (DNS resolution just stopped working once npm install started, but killing it would make the problem go away), which is how I got here this time.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked