Porting libnetwork fixes

[aboch]

Brings to 1.12.x the fixes for the following issues:

• [1.12.3] dockerd: panic: runtime error: invalid memory address or nil pointer dereference #28112
• docker daemon panic with swarmkit and lots of services(300+) #28465
• dockerd crashes (panics) when passing --link-local-ip to docker run. #28606
• docker server panic , when network plugin return nil StaticRoutes #28630
• panic: runtime error: slice bounds out of range #25184
• [1.12.3] Swarm node crash "fatal error: concurrent map read and map write" #28697
• panic: concurrent map read and map write #28845
• Execute syscall.NETLINK_XFRM from inside a container #27188
• The --ipv6 flag to docker network create is broken for remote plugins in Docker 1.12.2 #28055
• Daemon hangs after starting and stopping many containers #26251
• Error response from daemon / Docker engine panic: leave after shutdown docker/for-mac#849
• Docker 1.12.3 swarm subnetwork --opt encrypted docker/for-mac#888
• netlink calls should have timeouts libnetwork#1474

[vieux]

@aboch looks like issues

[aboch]

Not sure about vendor failure:

00:33:28 hack/dind: line 29: /go/src/github.com/docker/docker/hack/validate/vendor: No such file or directory
00:33:29 Build step 'Execute shell' marked build as failure

It looks like it is looking for the validate directory that was added for the new vendoring, which is not present in 1.12.x branch.

[anusha-ragunathan]

experimental is failing because in 1.12, plugins are still in experimental.

The media type for plugins was changed in 1.13 to application/vnd.docker.plugin.v1+json https://github.com/docker/distribution/blob/4ad885a6725bfca9ea00bd87341e367c0f919560/manifest/schema2/manifest.go#L21. As a result, the sample plugin we use in integration tests tiborvass/no-remove was re-pushed to Hub using this new mediatype.

That's why the tests are failing. We have 2 options:

1. Backport the plugin mediatype changes to 1.12.x
2. Repush a 1.12 version of tiborvass/no-remove that has the old mediatype and update the tests to use it.

I prefer (1)

[anusha-ragunathan]

For the plugin mediatype changes, we need 970b23d

[anusha-ragunathan]

An easier option is to simply add the new media type here https://github.com/aboch/docker/blob/76d2f1f29bbc11c92ab20489fc5eee2593b6d879/plugin/distribution/types.go#L16. Then use both the new and the old type for checking in https://github.com/aboch/docker/blob/76d2f1f29bbc11c92ab20489fc5eee2593b6d879/plugin/distribution/pull.go#L146

[vieux]

@johnstep can you please try this one, since there is no CI for 1.12.x on Windows

[johnstep]

LGTM on Windows. It built successfully on Windows (for Windows) and all unit tests passed. I rebuilt and tested again with the resulting binaries, and that succeeded.

[anusha-ragunathan]

If this check is updated as follows, we should be good:

if m.Config.MediaType != MediaTypeConfig &&
   m.Config.MediaType != "application/vnd.docker.plugin.v0+json" &&
   m.Config.MediaType != "application/vnd.docker.plugin.image.v0+json"{
...
}

[anusha-ragunathan]

All experimental tests are failing due to the known plugin mediatype issue. It can be fixed by updating the code to accommodate to the expected media type of the sample plugin. Or the sample plugin can be re-pushed by @tiborvass

In either case, this PR can be merged, since the failing tests are unrelated to the PR.

[vieux]

LGTM ping @LK4D4 @tonistiigi

[aboch]

@anusha-ragunathan Thanks for your help in trying to solve the experimental failure on 1.12.x.
Given it is currently broken, I feel it would be more efficient if you guys address the plugin failure on experimental ci issue via a separate PR. Will also give you more control over what to change. After that is merged we can rebase our vendoring (libnetwork/swarmkit) PRs .
I went ahead and removed the uncomplete media type changes commit from this PR.

[anusha-ragunathan]

@aboch : Yessir! #29089

[vieux]

@tonistiigi windows is expected to fail

[mavenugo]

#29089 handles the experimental failures.

LGTM