swarm classic ping resolves but fail in overlay network

[antoinetran]

Dear all and @fcrisciani ,

Description:
Our project started a new clean deployment of Docker Swarm Classic and we have a network connectivity issue. The issue impacts all containers in the same node. They cannot ping any other containers in other node in one of a overlay network. (In other node, containers can ping themselves even if they are in different nodes.) The ping fails with unreacheable error.

Environment:

• RedHat 7.4.1708 / docker-ce-1.12.0-ce / 3 Swarm managers (Swarm image 1.2.8) / 3 zookeeper 3.4.10 as cluster-store
• storage: overlayfs2

Expected: the ping works.

Logs: I have create an archive with all logs (var/log/message, journalctl -u docker, docker logs SwarmNode, SwarmManager, zookeepers), all of that is a 13MB tgz. Link: http://www.filedropper.com/logsdocker @fcrisciani is-it possible somehow for you to look at them after I send
that to you securely (email, form, etc.)?

Here is an extract of the docker logs of the machine with the issue:
-- Logs begin at Thu 2018-03-15 15:30:51 UTC, end at Thu 2018-03-15 15:54:06 UTC. --
Mar 15 15:31:05 fl2psto-s00.l2pf.fac.mtg systemd[1]: Starting Docker Application Container Engine...
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06.000508239Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06.004742224Z" level=info msg="libcontainerd: started new docker-containerd process" pid=2307
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="starting containerd" module=containerd revision=89623f28b87a6004d4b785663257362d1658a729 version=v1.0.0
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="setting subreaper..." module=containerd
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="changing OOM score to -500" module=containerd
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." module=containerd type=io.containerd.content.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." module=containerd type=io.containerd.snapshotter.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." module=containerd type=io.containerd.snapshotter.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." module=containerd type=io.containerd.metadata.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." module=containerd type=io.containerd.differ.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." module=containerd type=io.containerd.gc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." module=containerd type=io.containerd.monitor.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." module=containerd type=io.containerd.runtime.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." module=containerd type=io.containerd.grpc.v1
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06Z" level=info msg="containerd successfully booted in 0.009304s" module=containerd
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06.232727566Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06.673057991Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06.675000365Z" level=info msg="Initializing discovery without TLS"
Mar 15 15:31:06 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:06.678650322Z" level=info msg="Loading containers: start."
Mar 15 15:31:07 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2psto-s01:2181: dial tcp: i/o timeout
Mar 15 15:31:07 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2pdck-s00:2181: dial tcp: i/o timeout
Mar 15 15:31:07 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2pdck-s00:2181: dial tcp: lookup fl2pdck-s00 on 10.4.117.51:53: dial udp 10.4.117.51:53: i/o timeout
Mar 15 15:31:08 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2ppro-s01:2181: dial tcp: i/o timeout
Mar 15 15:31:08 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2ppro-s01:2181: dial tcp: lookup fl2ppro-s01 on 10.4.117.51:53: dial udp 10.4.117.51:53: i/o timeout
Mar 15 15:31:08 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2psto-s01:2181: dial tcp: i/o timeout
Mar 15 15:31:10 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2pdck-s00:2181: dial tcp: i/o timeout
Mar 15 15:31:10 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2psto-s01:2181: dial tcp: i/o timeout
Mar 15 15:31:10 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2psto-s01:2181: dial tcp: lookup fl2psto-s01 on 10.4.117.51:53: dial udp 10.4.117.51:53: i/o timeout
Mar 15 15:31:10 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2ppro-s01:2181: dial tcp: i/o timeout
Mar 15 15:31:10 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2ppro-s01:2181: dial tcp: lookup fl2ppro-s01 on 10.4.117.51:53: dial udp 10.4.117.51:53: i/o timeout
Mar 15 15:31:10 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:10.727924387Z" level=error msg="discovery error: zk: could not connect to a server"
Mar 15 15:31:11 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2pdck-s00:2181: dial tcp: i/o timeout
Mar 15 15:31:11 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: Failed to connect to fl2pdck-s00:2181: dial tcp: lookup fl2pdck-s00 on 10.4.117.51:53: dial udp 10.4.117.51:53: i/o timeout
Mar 15 15:31:11 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:11.979190144Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.480947458Z" level=warning msg="error locating sandbox id 17eb0e52852b2d4bb260427b889f3bb0d012d1aa7218f3c3f0e67cfbc839cfd8: sandbox 17eb0e52852b2d4bb260427b889f3bb0d012d1aa7218f3c3f0e67cfbc839cfd8 not found"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.482298586Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/edddc3c6146cdf54250754415923a219cf900a78c57cb98c62845b6f05a87963/shim.sock" debug=false module="containerd/tasks" pid=2748
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.498133595Z" level=error msg="220e92d91f2bc98281dedf2695d08b6336c293612231b55f3dc2c13c899b10bf cleanup: failed to delete container from containerd: no such container"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.498181727Z" level=error msg="Failed to start container 220e92d91f2bc98281dedf2695d08b6336c293612231b55f3dc2c13c899b10bf: endpoint with name fac_kafka-broker-1_1 already exists in network fac_internal_data"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.553085458Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/5d973fd10579e7be1b8c006ab54b3b7b04fdc8403260c75e163bebc30c2f5886/shim.sock" debug=false module="containerd/tasks" pid=2772
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.734029798Z" level=warning msg="unknown container" container=5d973fd10579e7be1b8c006ab54b3b7b04fdc8403260c75e163bebc30c2f5886 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.734361974Z" level=error msg="Failed to deserialize netlink ndmsg: invalid argument"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.769417779Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.770130607Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.782303682Z" level=warning msg="unknown container" container=5d973fd10579e7be1b8c006ab54b3b7b04fdc8403260c75e163bebc30c2f5886 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.827370314Z" level=warning msg="unknown container" container=edddc3c6146cdf54250754415923a219cf900a78c57cb98c62845b6f05a87963 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.871208039Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.880504225Z" level=warning msg="unknown container" container=edddc3c6146cdf54250754415923a219cf900a78c57cb98c62845b6f05a87963 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.948263567Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.949088105Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:12 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:12.965599533Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:13 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:13.007926683Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Mar 15 15:31:13 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:13.010082330Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Mar 15 15:31:13 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:13.014621047Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Mar 15 15:31:13 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:13Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/1819f0951158e08891af6e17d9cbef8b6ed6cd2f01528cde104f27b2233921b0/shim.sock" debug=false module="containerd/tasks" pid=2954
Mar 15 15:31:13 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:13Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/46c449c5f170f92bf75745bf2a6651b6a5cdba0c7af32a56811939239ae1a636/shim.sock" debug=false module="containerd/tasks" pid=2956
Mar 15 15:31:13 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:13Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/851ae4dca73944a94bc70223fd65710353bc429ab698dbe031640e23a49643ba/shim.sock" debug=false module="containerd/tasks" pid=2966
Mar 15 15:31:13 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:13.482129022Z" level=warning msg="unknown container" container=1819f0951158e08891af6e17d9cbef8b6ed6cd2f01528cde104f27b2233921b0 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:13 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:13.525360273Z" level=warning msg="unknown container" container=1819f0951158e08891af6e17d9cbef8b6ed6cd2f01528cde104f27b2233921b0 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:14 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:14.564350076Z" level=warning msg="unknown container" container=46c449c5f170f92bf75745bf2a6651b6a5cdba0c7af32a56811939239ae1a636 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:14 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:14.572168306Z" level=warning msg="unknown container" container=851ae4dca73944a94bc70223fd65710353bc429ab698dbe031640e23a49643ba module=libcontainerd namespace=plugins.moby
Mar 15 15:31:14 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:14.608201490Z" level=warning msg="unknown container" container=46c449c5f170f92bf75745bf2a6651b6a5cdba0c7af32a56811939239ae1a636 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:14 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:14.617501579Z" level=warning msg="unknown container" container=851ae4dca73944a94bc70223fd65710353bc429ab698dbe031640e23a49643ba module=libcontainerd namespace=plugins.moby
Mar 15 15:31:15 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:15.423931572Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Mar 15 15:31:15 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:15Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/6a02c131d1142ebb4aa047a3ffa810b0c38ac687e7810b8ff48431ce43dab4b2/shim.sock" debug=false module="containerd/tasks" pid=3335
Mar 15 15:31:15 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:15.898992743Z" level=warning msg="unknown container" container=6a02c131d1142ebb4aa047a3ffa810b0c38ac687e7810b8ff48431ce43dab4b2 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:15 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:15.941339073Z" level=warning msg="unknown container" container=6a02c131d1142ebb4aa047a3ffa810b0c38ac687e7810b8ff48431ce43dab4b2 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:15 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:15.966401535Z" level=info msg="Loading containers: done."
Mar 15 15:31:16 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:16.051985782Z" level=info msg="Docker daemon" commit=c97c6d6 graphdriver(s)=overlay2 version=17.12.0-ce
Mar 15 15:31:16 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:16.052573326Z" level=info msg="Daemon has completed initialization"
Mar 15 15:31:16 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:16.065777792Z" level=info msg="API listen on 10.4.117.20:2375"
Mar 15 15:31:16 fl2psto-s00.l2pf.fac.mtg systemd[1]: Started Docker Application Container Engine.
Mar 15 15:31:16 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:16.065825523Z" level=info msg="API listen on /var/run/docker.sock"
Mar 15 15:31:22 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:22.095756778Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:22 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:22.095865595Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:23 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:23.095860037Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:23 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:23.096795888Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:24 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:24.098780152Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:24 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:24.098866297Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:25 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:25.100858024Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:42 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:42.293814794Z" level=warning msg="unknown container" container=1819f0951158e08891af6e17d9cbef8b6ed6cd2f01528cde104f27b2233921b0 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:42 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:42.295488763Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:42 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:42.295550067Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:31:43 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:43.296787190Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
..........
Mar 15 15:31:43 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:43.933957125Z" level=warning msg="unknown container" container=1819f0951158e08891af6e17d9cbef8b6ed6cd2f01528cde104f27b2233921b0 module=libcontainerd namespace=plugins.moby
Mar 15 15:31:44 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:31:44.298788081Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Mar 15 15:53:32 fl2psto-s00.l2pf.fac.mtg dockerd[2187]: time="2018-03-15T15:53:32.106767133Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"

[antoinetran]

I can add that this issue does not seems related to #962 because the reverse ping does not solve the situation.

[antoinetran]

All machines ping between themselves and all SwarmManager SwarmNode ports are open.

There is one hypothesis we are not sure if this is connected: the ping of machine are slow in short name, and fast in FQDN.

[antoinetran]

We tried numerous reboot and reset-swarm (a script that removes all containers in all machines), the issue was still here.

We upgraded in urgency docker-ce from 17.12.0 to 17.12.1 and this seems to resolve the situation. No more ping issue. In /var/log/messages, we do not have the "Failed to deserialize netlink ndmsg: Link not found" pattern anymore, that is fixed in this version. But that may be unrelated. For people having issue with 17.12.0, I suggest to upgrade to latest stable.