Support for dual-stack Kubernetes cluster

[carpenike]

Is your feature request related to a problem? Please describe.
Currently, the parameter inputs (--cluster-cidr, --serice-cidr, etc) do not support more than one entry and that entry must be an IPv4 range. Enabling dual-stack clusters (https://kubernetes.io/docs/concepts/services-networking/dual-stack/) is currently not possible.

Describe the solution you'd like
Enable support for dual-stack kubernetes clusters

Describe alternatives you've considered
N/A

Additional context

Starting k3s with below settings results in a failure on invalid CIDR:

   k3s server \
        '--tls-san' \
        '{{ k3s_master }}' \
        '--no-deploy' \
        'servicelb' \
        '--no-deploy' \
        'traefik' \
        '--docker' \
        '--flannel-backend=none' \
        '--default-local-storage-path=/dev/shm' \
        '--cluster-cidr=10.244.0.0/16,<RANGE>:c101:5810::/64' \
        '--service-cidr=10.43.0.0/16,<RANGE>:c101:5811::/64' \

[carpenike]

Continuation of #284

[pdf]

I have a bare-minimum approach working locally for allowing dual-stack, but it comes with the following caveats:

• IPv4 is enforced as the default protocol (order of cluster-cidr/service-cidr is ignored)
• Cluster components run only on IPv4
• Flannel is automatically disabled when enabling dual-stack, as it is not compatible, so users must bring their own CNI after server start

The first two caveats can likely be removed with some additional effort (I have this mostly working), which should also pave the way for IPv6-only, however there are some concerning pieces of code in the repo, primarily:

https://github.com/rancher/k3s/blob/56a083c812604f8edeac410714aaf81a7f10e295/pkg/agent/netpol/network_policy_controller.go

and friends, imported from kube-router, which currently has no support for IPv6. It's not clear to me what this code is used for.

I've also not yet validated that the k3s loadbalancer, or the traefik ingress function correctly for IPv6 services/endpoints, though I suspect the latter should work fine.

[ieugen]

I would love to deploy k3s with IPV6 in my home (my ISP offers IPV6).
@pdf : Could you please share how you made k3s work with IPV6?

[pdf]

Unfortunately I've not been able to prioritise finishing this yet - the change is not in a state where I would expect everything to function correctly, and there are some issues with IPv6 all the way up the stack currently.

[optix2000]

I have a bare-minimum approach working locally for allowing dual-stack, but it comes with the following caveats:

• IPv4 is enforced as the default protocol (order of cluster-cidr/service-cidr is ignored)
• Cluster components run only on IPv4
• Flannel is automatically disabled when enabling dual-stack, as it is not compatible, so users must bring their own CNI after server start

@pdf Any chance you can release the patchset or start a PR for this? I'd be interested in furthering this along.

[unixfox]

Dual stack is coming to flannel: flannel-io/flannel#1398

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[DasSkelett]

IPv6 support for Flannel got merged: flannel-io/flannel#1448

So that blocker is out of the way, I guess now it's on k3s to make something out of it.

[samip5]

Currently when trying to get kube-dns edited with IPv6 support, and using Calico as CNI it results in errors as the cluster is not configured for dual stack services according to kubectl.

kubectl svc edit kube-dns -n kube-system and changing ipFamilyPolicy to RequireDualStack results in:

# services "kube-dns" was not valid:
# * spec.ipFamilyPolicy: Invalid value: "RequireDualStack": Cluster is not configured for dual stack services

at the top of the edited file which it will not let me commit with that change.

[brandond]

@samip5 are you using k3s, or a kubeadm-managed cluster? If the latter, I'm not sure you're in the right place.

[samip5]

@samip5 are you using k3s, or a kubeadm-managed cluster? If the latter, I'm not sure you're in the right place.

I'm using k3s, what made you thought that it would be the latter?

[brandond]

Well you said

the cluster is not configured for dual stack services according to kubeadm.

but then edited it before replying, so you clearly knew why I was asking...

[samip5]

Well you said

the cluster is not configured for dual stack services according to kubeadm.

but then edited it before replying, so you clearly knew why I was asking...

Yes, because to me kubectl and kubeadm were kind of the same thing, which appears to be not true so I wasn't sure which tool made the message in reality when it came up when trying to edit the svc.

[brandond]

If you want to enable your cluster for DualStack operation, you need to be using K3s 1.21+ and provide your own CNI plugin, node IPv6 addresses, and IPv6 pod/service CIDRs. There is some conversation over at #3212 that you could use as a starting place.

[samip5]

If you want to enable your cluster for DualStack operation, you need to be using K3s 1.21+ and provide your own CNI plugin, node IPv6 addresses, and IPv6 pod/service CIDRs. There is some conversation over at #3212 that you could use as a starting place.

By default, my cluster got an IPv6 pod cidr even though the k3s server code doesn't seem to provision it but rather kubernetes itself as that's the default. Using k3s version 1.21.3.

Default config (with Calico CNI) currently results in something similar to this:

$ k get nodes -o yaml |grep -EA6 'spec:|addresses:'
  spec:
    podCIDR: 10.244.7.0/24
    podCIDRs:
    - 10.244.7.0/24
    - fddf:f7bc:9670:7::/64
    providerID: k3s://k8s-worker6
  status:
    addresses:
    - address: 192.168.2.45
      type: InternalIP
    - address: k8s-worker6
      type: Hostname
    allocatable:
      cpu: "4"
--
  spec:
    podCIDR: 10.244.5.0/24
    podCIDRs:
    - 10.244.5.0/24
    - fddf:f7bc:9670:5::/64
    providerID: k3s://k8s-worker5
  status:
    addresses:
    - address: 192.168.2.69
      type: InternalIP
    - address: k8s-worker5
      type: Hostname
    allocatable:
      cpu: "4"
--
  spec:
    podCIDR: 10.244.3.0/24
    podCIDRs:
    - 10.244.3.0/24
    - fddf:f7bc:9670:3::/64
    providerID: k3s://k8s-worker3
  status:
    addresses:
    - address: 192.168.2.57
      type: InternalIP
    - address: k8s-worker3
      type: Hostname
    allocatable:
      cpu: "4"
--
  spec:
    podCIDR: 10.244.0.0/24
    podCIDRs:
    - 10.244.0.0/24
    - fddf:f7bc:9670::/64
    providerID: k3s://kubemaster
    taints:
--
    addresses:
    - address: 192.168.2.9
      type: InternalIP
    - address: kubemaster
      type: Hostname
    allocatable:
      cpu: "4"
--
  spec:
    podCIDR: 10.244.1.0/24
    podCIDRs:
    - 10.244.1.0/24
    - fddf:f7bc:9670:1::/64
    providerID: k3s://k8s-worker1
  status:
    addresses:
    - address: 192.168.2.62
      type: InternalIP
    - address: k8s-worker1
      type: Hostname
    allocatable:
      cpu: "4"
--
  spec:
    podCIDR: 10.244.2.0/24
    podCIDRs:
    - 10.244.2.0/24
    - fddf:f7bc:9670:2::/64
    providerID: k3s://k8s-worker2
  status:
    addresses:
    - address: 192.168.2.63
      type: InternalIP
    - address: k8s-worker2
      type: Hostname
    allocatable:
      cpu: "4"
--
  spec:
    podCIDR: 10.244.4.0/24
    podCIDRs:
    - 10.244.4.0/24
    - fddf:f7bc:9670:4::/64
    providerID: k3s://k8s-worker4
  status:
    addresses:
    - address: 192.168.2.64
      type: InternalIP
    - address: k8s-worker4
      type: Hostname
    allocatable:
      cpu: "2"

Why is there no documentation for how to properly provision it atm with full stack support?

Can I add the IPv6 node addresses after the agent was provisioned to just the systemd args or does it need to be re-installed?

[onedr0p]

Why is there no documentation for how to properly provision it atm with full stack support?

Because dual stack is beta and not widely used. Moreover k3s defaults to the flannel CNI which does not support dual stack.

See flannel-io/flannel#248

[samip5]

Why is there no documentation for how to properly provision it atm with full stack support?

Because dual stack is beta and not widely used. Moreover k3s defaults to the flannel CNI which does not support dual stack.

See flannel-io/flannel#248

Flannel in fact does support dual stack which was introduced recently.

Please see flannel-io/flannel#1448

[onedr0p]

Feel free to open a PR to the docs on how to make it work then! It appears dual stack is only supported using VXLAN on Flannel.

[brandond]

I believe the CIDRs you're seeing all come from Calico's IPAM.

For Flannel, as you said, IPv6 support was only introduced recently. You'll note that there has not been a Flannel release that includes it yet, it's only available on the master branch. We're well aware of the progress on that issue; @manuelbuil is a member of our team and is actively working on end-to-end dual stack support, but it is at the moment a do-it-yourself affair.

[manuelbuil]

Here is the plan for this:

• Integrate latest flanneld (with dual-stack feature) into k3s ==> Add dual-stack support on flannel #3906
• Move the flannel binary that k3s uses to a version that contains the dual-stack feature (e.g. 0.9.1). First PR there Move to v0.9.1 rancher/plugins#5
• Adapt netpool to dual-stack https://github.com/k3s-io/k3s/tree/master/pkg/agent/netpol
• Dual-stack support in klipper-lb and lb controller ==> Use the newer klipper-lb image #4029
• Add tests to k3s ==> Add dual-stack support on flannel #3906
• Document everything ==> Include docs about dual-stack config rancher/docs#3620

[vadorovsky]

There is also a need for adding dual-stack support in kube-router (and then our fork / copied code). I'm on it.

Upstream issue: cloudnativelabs/kube-router#307

[manuelbuil]

There is also a need for adding dual-stack support in kube-router (and then our fork / copied code). I'm on it.

Upstream issue: cloudnativelabs/kube-router#307

Kube-router still does not support dual-stack. Until that, network policies need to be disabled when deploying k3s with the default CNI plugin (flannel)

[ShylajaDevadiga]

$ k3s -v
k3s version v1.21.7-rc1+k3s1
$ kubectl get svc -A |grep my-service
default       my-service-v4             LoadBalancer   10.43.179.62           192.168.16.254,192.168.25.10                                                                                 8081:32209/TCP               7m27s
default       my-service-v6             LoadBalancer   2001:cafe:42:1::4c67   2600:1f1c:ab4:ee48:3675:b33c:4e7c:e3b1,2600:1f1c:ab4:ee48:8999:b1bd:9950:5358                                8082:30605/TCP               7m27s
default       my-service-prefer-dual    LoadBalancer   10.43.81.16            192.168.16.254,192.168.25.10,2600:1f1c:ab4:ee48:3675:b33c:4e7c:e3b1,2600:1f1c:ab4:ee48:8999:b1bd:9950:5358   8084:31195/TCP               7m26s
default       my-service-require-dual   LoadBalancer   10.43.16.202           192.168.16.254,192.168.25.10,2600:1f1c:ab4:ee48:3675:b33c:4e7c:e3b1,2600:1f1c:ab4:ee48:8999:b1bd:9950:5358   8083:30214/TCP               7m27s

$ curl http://192.168.16.254:8084
<html><body><h1>It works!</h1></body></html>
$ curl http://192.168.25.10:8084
<html><body><h1>It works!</h1></body></html>
$ curl http://[2600:1f1c:ab4:ee48:3675:b33c:4e7c:e3b1]:8084
<html><body><h1>It works!</h1></body></html>
$ curl http://[2600:1f1c:ab4:ee48:8999:b1bd:9950:5358]:8084
<html><body><h1>It works!</h1></body></html>

Validated two ips ipv4 and ipv6 are available within the pod

3: eth0@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP group default 
    link/ether aa:fa:7b:19:32:64 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.42.0.15/24 brd 10.42.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2001:cafe:42::f/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::a8fa:7bff:fe19:3264/64 scope link 
       valid_lft forever preferred_lft forever