Overlay Network : Unable ping container from one host to another

[sujeetkp]

I am trying to create an overlay network without swarm.

I have 3 hosts. (2 ubuntu and 1 centos)

On one of the ubuntu hosts (ubuntu host 1) I am running the consul container.

docker run -d -p 8500:8500 -h consul --name consul progrium/consul -server -bootstrap

After that I have started the docker daemons on the other ubuntu host (ubuntu host 2) and centos host with the below command

sudo /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-advertise docker0:2375 --cluster-store consul://<public-ip-ubuntu host 1>:8500

Then I created the overlay network as below.

docker network create -d overlay --subnet=192.168.3.0/24 my-overlay

I can see my overlay network on both the hosts.

After that I created containerA on ubuntu host 2 and containerB on centos.

docker run -itd --name containerA --net my-overlay busybox

docker run -itd --name containerB --net my-overlay busybox

But I am not able to ping one container from the other.

root@instance-1:~# docker exec containerA ping -w 5 containerB
PING containerB (192.168.3.3): 56 data bytes

--- containerB ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
root@instance-1:~#

[root@docker-2 ~]# docker exec containerB ping -w 5 containerA
PING containerA (192.168.3.2): 56 data bytes

--- containerA ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
You have new mail in /var/spool/mail/root

root@instance-1:~# docker inspect containerA
[
{
"Id": "c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e",
"Created": "2017-12-28T17:42:37.47345131Z",
"Path": "sh",
"Args": [],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3871,
"ExitCode": 0,
"Error": "",
"StartedAt": "2017-12-28T17:57:29.578002102Z",
"FinishedAt": "2017-12-28T17:57:06.102289772Z"
},
"Image": "sha256:6ad733544a6317992a6fac4eb19fe1df577d4dec7529efec28a5bd0edad0fd30",
"ResolvConfPath": "/var/lib/docker/containers/c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e/hostname",
"HostsPath": "/var/lib/docker/containers/c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e/hosts",
"LogPath": "/var/lib/docker/containers/c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e/c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e-json.log",
"Name": "/containerA",
"RestartCount": 0,
"Driver": "aufs",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "my-overlay",
"PortBindings": {},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Data": null,
"Name": "aufs"
},
"Mounts": [],
"Config": {
"Hostname": "c34854d53ce7",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": true,
"OpenStdin": true,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"sh"
],
"ArgsEscaped": true,
"Image": "busybox",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "6849fbb88356ea9dae5a3a756b187277de535d29ea2f20d1291dfcdc37a456de",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/6849fbb88356",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"my-overlay": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"c34854d53ce7"
],
"NetworkID": "cc0db79b09b56e2855799debeb08b17d4c570a63f940fa3bea84212695f040e6",
"EndpointID": "b178364726482e9444a1dff3acba64d3b241eeab989647610a4f591a4ddbad28",
"Gateway": "",
"IPAddress": "192.168.3.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:c0:a8:03:02",
"DriverOpts": null
}
}
}
}
]

[root@docker-2 ~]# docker inspect containerB
[
{
"Id": "29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f",
"Created": "2017-12-28T17:33:49.881385212Z",
"Path": "sh",
"Args": [],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 15263,
"ExitCode": 0,
"Error": "",
"StartedAt": "2017-12-28T17:59:22.926608023Z",
"FinishedAt": "2017-12-28T17:59:05.940642817Z"
},
"Image": "sha256:6ad733544a6317992a6fac4eb19fe1df577d4dec7529efec28a5bd0edad0fd30",
"ResolvConfPath": "/var/lib/docker/containers/29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f/hostname",
"HostsPath": "/var/lib/docker/containers/29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f/hosts",
"LogPath": "/var/lib/docker/containers/29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f/29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f-json.log",
"Name": "/containerB",
"RestartCount": 0,
"Driver": "devicemapper",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "my-overlay",
"PortBindings": {},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Data": {
"DeviceId": "30",
"DeviceName": "docker-8:1-25190669-966cdbfb8d46b90b0408005bff08bfb3b3aad6e458846eec5c23737e28778f94",
"DeviceSize": "10737418240"
},
"Name": "devicemapper"
},
"Mounts": [],
"Config": {
"Hostname": "29044717920e",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": true,
"OpenStdin": true,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"sh"
],
"ArgsEscaped": true,
"Image": "busybox",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "ef3cad5d7be81bdd78795d5c9fc257fde14e0a3bbc63f38800b3e5226cabdf1f",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/ef3cad5d7be8",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"my-overlay": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"29044717920e"
],
"NetworkID": "cc0db79b09b56e2855799debeb08b17d4c570a63f940fa3bea84212695f040e6",
"EndpointID": "abdf2d84e506333fab8f6a5b65704b720497aa121965f1a4956e8a852baf18fb",
"Gateway": "",
"IPAddress": "192.168.3.3",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:c0:a8:03:03",
"DriverOpts": null
}
}
}
}
]
You have new mail in /var/spool/mail/root
[root@docker-2 ~]#

root@instance-1:~# docker network inspect my-overlay
[
{
"Name": "my-overlay",
"Id": "cc0db79b09b56e2855799debeb08b17d4c570a63f940fa3bea84212695f040e6",
"Created": "2017-12-28T16:59:51.149732661Z",
"Scope": "global",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "192.168.3.0/24"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e": {
"Name": "containerA",
"EndpointID": "b178364726482e9444a1dff3acba64d3b241eeab989647610a4f591a4ddbad28",
"MacAddress": "02:42:c0:a8:03:02",
"IPv4Address": "192.168.3.2/24",
"IPv6Address": ""
},
"ep-abdf2d84e506333fab8f6a5b65704b720497aa121965f1a4956e8a852baf18fb": {
"Name": "containerB",
"EndpointID": "abdf2d84e506333fab8f6a5b65704b720497aa121965f1a4956e8a852baf18fb",
"MacAddress": "02:42:c0:a8:03:03",
"IPv4Address": "192.168.3.3/24",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]

root@ubuntu-vm-1404:~# docker info
Containers: 20
Running: 1
Paused: 0
Stopped: 19
Images: 28
Server Version: 17.06.2-ee-5
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 80
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 4.4.0-101-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.796GiB
Name: ubuntu-vm-1404
ID: HRHN:J4O2:IBNT:Z65D:HLX6:6DHC:RVJD:BISJ:AHRK:SWAB:WZMN:YUDK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
provider=generic
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
root@ubuntu-vm-1404:~#

[Vacant0mens]

Have you checked netstat -ano to see if port 4789/udp (overlay container communication) is open/listening?
For me, it seems to be a problem with custom overlay networks not opening that port properly.

ps - adding three ` (back-tick's) before and after your json/terminal output (block code) and one back-tick before and after command-line commands (in-line code) will make them more readable.

[Deequeue]

Have any workarounds / fixes been posted for this issue as yet?

I am also experiencing the same bug in testing, using the stable apt repo for ubuntu.

netstat -ano shows 4789 udp in a "blank" non listening state on both hosts being tested.

[ghost]

@Vacant0mens did you get this fixed?

[Vacant0mens]

I didn't. We stopped using Docker temporarily until some of these network issues get fixed.

I've heard that Windows Server 1709, and 1803 with Docker 18.03 fixes a lot of the networking issues but I haven't yet been able to confirm it myself.

[tomery2000]

Has this problem been solved?
I managed to get udp port 4789 open by connecting to my cluster using this command

sudo /usr/bin/dockerd -H tcp://0.0.0.0:4789 -H unix:///var/run/docker.sock --cluster-advertise enp0s3:4789 --cluster-store consul://10.0.2.157:8500

I can see the network as well as add containers to it just like but still cannot ping containers across the network, just like OP.

Might be worth noting, I am behind a corporate proxy. Not sure about anyone else.

[tomery2000]

UPDATE

I managed to solve the issue. Not sure if anyone else has or if we were suffering from different things.
I had an error which I had decided to ignore saying something along the lines of "reeeee its got the same name reeeee" turns out this was pretty important and what was breaking it.
To solve I ran "nmtui" which gives you a nice blue screen and a menu, I then went to the bottom option and changed the hostname of one of my machines.
If machine1 is host and machine 2&3 are the two on the network I changed one of 2 or 3. Anything will do, as long as they are not the same.
This solved the issue, if I worded it badly just say and I will try again.

Link to video I used. Only first minute or two are required.
https://www.youtube.com/watch?v=i23D7wNCuuc

[ghost]

I got my problems solved by running 18.06.1-ce on both Linux (Ubuntu/Debian) and Windows (server 1803) on physical servers. For Windows, you have to compile 18.06.1-ce yourself.

[ArtoriaRen]

UPDATE

I managed to solve the issue. Not sure if anyone else has or if we were suffering from different things.
I had an error which I had decided to ignore saying something along the lines of "reeeee its got the same name reeeee" turns out this was pretty important and what was breaking it.
To solve I ran "nmtui" which gives you a nice blue screen and a menu, I then went to the bottom option and changed the hostname of one of my machines.
If machine1 is host and machine 2&3 are the two on the network I changed one of 2 or 3. Anything will do, as long as they are not the same.
This solved the issue, if I worded it badly just say and I will try again.

Link to video I used. Only first minute or two are required.
https://www.youtube.com/watch?v=i23D7wNCuuc

This also solves my problem! Thank you!