WIP: Make it possible to configure IPv6 with Calico

[ederst]

This will enable users to configure IPv6 for the pod network with Calico when setting the IPv6AutodetectionMethod to something else than none.

The motivation behind this is to let users bring their own IPv6 net. For example in OpenStack adding an additional port with an IPv6 address can do this (depends on #13861, and also kubernetes-retired/etcdadm#317).

Limitations:

• Users have to care about adding an IPv6 to the host(s)
• Default IPv6 pool of Calico is used right now
• The config for if IsIpv6Only() is true has priority

[k8s-ci-robot]

Hi @ederst. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign johngmyers after the PR has been reviewed.
You can assign the PR to them by writing /assign @johngmyers in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hakman]

Hi @ederst. I appreciate the effort here, though IPv6 is quite a special topic. I would suggest that you join a Office Hours call so that you can discuss the intention and possible implementation options with the maintainers.

For background, in the past there was a debate about the scope of IPv6 support in kOps and the conclusion was that there was no plan to support DualStack or IPv6 NAT.

/cc @justinsb @johngmyers @olemarkus

[olemarkus]

It's somewhat unclear what changes in the cloud provider is needed for this and how that will look like.

I second what @hakman said around design principles though:

• kops should not support dualstack unless someone can give a (very) compelling reason for it. OpenStack is a bit different from AWS, but still ...
• kops should not support UKA ranges. IPv6 with ULA doesn't provide a while lot of benefits over IPv4 private addressing.

[k8s-ci-robot]

@ederst: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ederst]

My intentions with this are not to add IPv6 and/or Dual-Stack support to kOps directly.

So I don't want to add provisioning for the infrastructure and config to for K8s which would be needed to run the cluster on IPv6/Dual-Stack to kOps.

What I want to do is though, to make the managed Calico plugin of kOps to accept config so that Calico can provision IPv6 to pods. To bring IPv6 to the hosts - and thus make the setup work - would be up to the user, because this will not be handled by kOps.

For example, I imagined for our use case in OpenStack - which is simply to be able to talk to an IPv6 only service outside of our K8s clusters, so in theory we do not even need the pods to talk to eachother - to add the the IPv6 ports to the instances with some sort of simple program running on static pods on the nodes. But this is all happening outside of of kOps - well except for using the Hooks/Cloud-Init feature to deploy the static pods - and only relies on Calico being configured to support IPv6.

However, maybe introducing the config and changing the logic for the Calico addon in kOps to make this "BYOIPV6" happen is also out of scope and I should just use the CNI networking feature and deploy Calico with the config I need without changing the code of kOps at all. I just found it more convenient this way.

[olemarkus]

I think in this case, you'd just want to set the correct subnet type on the subnet structs and kOps should do the right thing.

[ederst]

I think in this case, you'd just want to set the correct subnet type on the subnet structs and kOps should do the right thing.

I am not sure if I understand you there a 100% but I don't think this will work, because:

• Currently IPv6 is restricted to the AWS cloud provider
• Currently the OpenStack provider only searches/creates subnets based on IPv4

To show what I mean, here is the code needed to make kOps and OpenStack work with a pre existing IPv6 subnet:

• master...infonova:kops:poc/os-ipv6-hack-based-on-master

Mind you that this code is based on my (scrapped) idea to make kOps provision dual stack ports to the instances, by adding multiple subnets to a port. In the manifest I still configured the K8s and node networks as IPv4, and needed to add the IPv6 range as an "additional network CIDR".

It might just be worth a try to adapt the code to work with IPv6 only. Unfortunately, I am fairly sure that IPv6 only will not work for the environment I have to deploy K8s in.

[olemarkus]

Yeah, I imagined some minor details would be needed for kOps to be aware of the subnet configuration, but that this wouldn't be too much work.

So dualstack node networking is fine, and that's what we usually do on AWS as well. dualstack pod networking, however, is more complicated. But as long as your setup has some form of NAT64/DNS64, ipv6-only pod networking should work.

[ederst]

The only problem we'll face is that our IPv6 addresses are in a different network, and I think in OS it is not possible to add subnets of different networks to one port. And with kOps it is not possible to specify more than one network and/or port.

At least that's what I think. Will check that on Monday.

[johngmyers]

The most logical way to get IPv6 on OpenStack would be to extend the OpenStack networking code much like the AWS code was. Like AWS, the pod network would be configured as IPv6-only and the nodes could be either dual-stack or IPv6-only. There would need to be NAT64 support somewhere in OpenStack.

[johngmyers]

You're saying the IPv4 and IPv6 addresses are from different NetworkIDs? Yeah, kOps expects there to be a single NetworkID/VPC-equivalent, as that is the way AWS models its networking.

Does OpenStack require IPv4 and IPv6 to be in separate NetworkIDs? In that case, you might need to make the IPv6 one the primary NetworkID, then have an OpenStack-specific secondary IPv4 NetworkID for making the nodes dual-stack.

[ederst]

You're saying the IPv4 and IPv6 addresses are from different NetworkIDs?

Yes, in our case.

Does OpenStack require IPv4 and IPv6 to be in separate NetworkIDs?

This is not required. We can provision "Tenant IPv6 Subnets" which we can put into the same Network(ID) like the IPv4s.

However, currently we have to use a provided IPv6 subnet (with a different NetworkID), because there is no NAT64 support. And if I understood our OS support correctly there won't be any in the near future.

So probably not some typical OpenStack setup.

This is actually the reason I scrapped my approach I posted earlier.

[johngmyers]

That seems a bit esoteric to support.

The IPv6 Subnets would have to route 64:ff9b::/96 to something that does NAT64. The NAT64 destination doesn't have to be in the same Network.

[ederst]

I think I misunderstood something here: What I meant is that NAT support - from ULA IPv6 range to a GUA:

At the current time OpenStack Networking does not provide any facility to support any flavor of NAT with IPv6. Unlike IPv4 there is no current embedded support for floating IPs with IPv6. It is assumed that the IPv6 addressing amongst the projects is using GUAs with no overlap across the projects.

Source: https://docs.openstack.org/neutron/latest/admin/config-ipv6.html#nat-floating-ips

So in our case we might need to provision a separate server/router for that.

I am still trying to understand how the 64:ff9b::/96 comes into play (IPv6 only cluster needs to talk to IPv4 services I guess) and what actually ends up being the Pod/ServiceCIDR of the cluster which seems to be used by Calico for IPAM when specifying ::/0 as NonMasqueradingCIDR like the doc suggests. Not sure if this is something I need to understand, but it really bugs me.

[johngmyers]

kOps IPv6 does not NAT from ULA to GUA. It relies on nodes each getting a GUA prefix, from which Kubernetes assigns GUAs to pods.

64:ff9b::/96 is the NAT64 prefix, used by things with IPv6 GUAs to NAT out to IPv4 destinations.

The kOps service network is fd00:5e4f:ce::/108. This is normally handled by kube-proxy, not the CNI.

[ederst]

I see, thanks for clarifying that for me.

So what I understand from that is, that the Pods will be in the same IPv6 network like the hosts and do not need NAT in any way. And since Pod networking is IPv6 only it will need some NAT64 to translate to IPv4.

Good, so now I know at least the direction I'd have to go to try to make this work for OpenStack. I will have a go on some quick and dirty prototype to see if that will even work, given that I have some spare time (no hard feelings if someone more experienced in that regard works on this too).

Anyhow, I will close this PR since the proposed change here will not be merged and the discussion should probably be moved to the OfficeHours and/or Slack in the mean time.

[johngmyers]

Your understanding is correct.