Moby ipv6 support

[amrmahdi]

Is this a request for help?:

Yes

---

Is this an ISSUE or FEATURE REQUEST? (choose one):

Issue

What version of acs-engine?:

0.25.2

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)

Kubernetes 1.10.8

What happened:

Moby container runtime does not seem to have ipv6 support. When upgrading acs-engine from 0.24.2 to 0.25.2 we noticed that container on the new pods does not have ipv6 support. Looking at the change log, it seems related to moby runtime change ?

What you expected to happen:

Containers have ipv6 support.

[amrmahdi]

@jackfrancis ideas ?

[jackfrancis]

@cpuguy83 @khenidak Does this ring a bell?

@amrmahdi in the meantime, you may still use docker-engine on new clusters in v0.25.2 by specifying the "aks-docker-engine" distro thusly:

(In all of your master and/or agent pools that need docker-engine)

    "masterProfile": {
      <etc...>
      "distro": "aks-docker-engine",
      <etc...>

    "agentPoolProfiles": [
      {
        <etc...>
        "distro": "aks-docker-engine",
        <etc...>

[cpuguy83]

Probably related to moby/moby#33099

I think if you run docker with --ipv6=true it will enable ipv6 in the container.

[jackfrancis]

Any reason not to add --ipv6=true to our bootstrap implementation for all clusters?

@amrmahdi Please let us know if that option unblocks your ipv6 workloads on moby-backed clusters, thanks!

[amrmahdi]

I’ve already tried modifing /etc/docker/daemon.json on individual nodes to add ```"ipv6": true’’’ but it didn’t help. Will try the distro option and let you know.

[cpuguy83]

Oh that makes sense, I guess this flag is specifically for docker networks created with docker network and not anything to do with the default networking that k8s uses... will do some more digging to see how others have solved this.
The issue is that docker does not disable the disable_ipv6 the option in the container (/proc/sys/net/ipv6/conf/default/disable_ipv6)

[cpuguy83]

What version of CNI are we using?

containernetworking/cni#531 (comment)

CNI Plugins v0.7.0 includes the disable_ipv6 sysctl workaround so hopefully Kubernetes IPv6 will be easier to get up and running now on newer Docker releases. Thanks for the swift response from the CNI maintainers!

[jackfrancis]

@cpuguy83 we're using v0.7.1

[amrmahdi]

@jackfrancis setting 'aks-docker-engine' resolved the ipv6 issue. But adding a new agent pool on this distro not does not work :(

Deployment failed. Correlation ID: 0158fb67-4811-46cd-8b47-b58e07fb738a. {
   "error": {
     "code": "PropertyChangeNotAllowed",
     "message": "Changing property 'customData' is not allowed.",
     "target": "customData"
   }
}

Any workarounds for this one ?

[jackfrancis]

Can you paste your process for adding an agent pool?

[amrmahdi]

1. Get the original apimodel.son generated by acs-engine.
2. Copy agentPoolProfiles[0] into the end of the array.
3. Update name, vmSize, count of the new pool appropriately to the new values.
4. Set distro of the new pool to 'aks-docker-engine'
5. Run acs-engine generate
6. Deploy the new generated template with incremental mode.

P.S. We also update ipAddressCount for all pools to match the original since acs-engine strips it out in the generated apimodel.json (possibly a bug)

[jackfrancis]

Is the distro of the original pool (step 1) "aks-docker-engine"?

[amrmahdi]

Yes.

[jackfrancis]

Then step 4 is "verify" not "set", right?

As this is not core-acs-engine functionality, we don't have any tests that validate that it works, just wanna make sure I understand what's not working.

[amrmahdi]

Well it does not matter in this case since we have made everything to use 'aks-docker-engine' to resolve this issue, and the other GPU issue we reported few days ago.

[jackfrancis]

So I suppose we can conclude (for reasons unknown) that for non-moby scenarios, manually editing the agent pool member array after cluster creation is broken.

[amrmahdi]

Yes it is. I'd say it is on the 'aks-docker-engine' distro. It could be related to the recent image version updates ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution. Note that acs-engine is deprecated--see https://github.com/Azure/aks-engine instead.