Add IPv6 support #348

aojea · 2019-02-25T14:22:58Z

This PR adds allows creating IPv6 Kubernetes clusters with kind and have in mind a future dual-stack implementation, considering for simplicity, only one address of each protocol.

It adds a new option ipFamily to the v1alpha3 API that allows choosing the IP family of the cluster.
To avoid issues with the different networking options the podSubnet and the serviceSubnet kubeadm values are predefined with the following values:

	Default PodSubnet          = "10.244.0.0/16"
	Default ServicesSubnet     = "10.96.0.0/12"
	Default PodSubnetIPv6      = "fd00:10:244::/64"
	Default ServicesSubnetIPv6 = "fd00:10:96::/112"

We can create a Kubernetes IPv6 cluster with the following config:

# necessary for conformance
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
  ipFamily: ipv6
nodes:
# the control plane node
- role: control-plane
- role: worker
- role: worker

Test results with IPv4 and IPv6

Conformance Tests

References:

Fixes #280

k8s-ci-robot · 2019-02-25T14:24:55Z

Hi @aojea. Thanks for your PR.

I'm waiting for a kubernetes-sigs or kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BenTheElder · 2019-02-25T16:59:45Z

/ok-to-test
I'd like to make this a config field in cluster wide networking re: #340

neolit123

thanks for the PR, i've added some generic comments.

images/base/Dockerfile

pkg/cluster/nodes/node.go

neolit123 · 2019-02-25T17:33:04Z

pkg/cluster/internal/kubeadm/config.go

@@ -74,11 +76,14 @@ clusterName: "{{.ClusterName}}"
 bootstrapTokens:
 - token: "{{ .Token }}"
 {{ if .ControlPlaneEndpoint -}}
-controlPlaneEndpoint: {{ .ControlPlaneEndpoint }}
+controlPlaneEndpoint: "{{ .ControlPlaneEndpoint }}"
 {{- end }}
 # we use a well know port for making the API server discoverable inside docker network. 
 # from the host machine such port will be accessible via a random local port instead.


this comment is about the port.

don't understand the comment, I had to quote the ControlPlaneEndpoint because it failed to create the JSON from the template using IPv6 addresses

i wanted to say that the comment # we use a well know port for making... is now above advertiseAddress, while it applies to the port bellow.

I think that the comment is ok, it's above the api section not above the port field, to explain that the apiendpoint uses a well know port.

pkg/cluster/internal/kubeadm/config.go

neolit123 · 2019-02-25T17:37:59Z

pkg/cluster/internal/create/kubeadm-config.go

+	if err != nil {
+		return errors.Wrap(err, "failed to get IP for bootsrap node")
+	}
+
 	// get the control plane endpoint, in case the cluster has an external load balancer in
 	// front of the control-plane nodes
 	controlPlaneEndpoint, err := getControlPlaneEndpoint(ec)


getControlPlaneEndpoint might need modifications based on IPv6.

have you tested an HA setup?

didn't test it, but will do.
I've already modified the function func getControlPlaneEndpoint( to support IPv6, check L135

@neolit123 results of an HA setup test

# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7b37bcbcf3b6 kindest/node:v1.13.3 "/usr/local/bin/entr…" 11 minutes ago Up 11 minutes kind-worker2 89fa42fd404e kindest/node:v1.13.3 "/usr/local/bin/entr…" 12 minutes ago Up 12 minutes kind-worker1 0d0caea990ca kindest/node:v1.13.3 "/usr/local/bin/entr…" 12 minutes ago Up 12 minutes 33453/tcp, 0.0.0.0:33453->6443/tcp kind-control-plane3 422bd7c52d53 kindest/node:v1.13.3 "/usr/local/bin/entr…" 13 minutes ago Up 12 minutes 34151/tcp, 0.0.0.0:34151->6443/tcp kind-control-plane2 780f8cde9d6d kindest/node:v1.13.3 "/usr/local/bin/entr…" 13 minutes ago Up 13 minutes 37927/tcp, 0.0.0.0:37927->6443/tcp kind-control-plane1 3f2e867b33b6 kindest/node:v1.13.3 "/usr/local/bin/entr…" 13 minutes ago Up 13 minutes 40189/tcp, 0.0.0.0:40189->6443/tcp kind-lb

# kubectl get endpoints --all-namespaces NAMESPACE NAME ENDPOINTS AGE default kubernetes [2001:db8:1::242:ac11:3]:6443,[2001:db8:1::242:ac11:4]:6443,[2001:db8:1::242:ac11:5]:6443 11m kube-system kube-controller-manager <none> 10m kube-system kube-dns 10.32.0.2:53,10.32.0.3:53,10.32.0.2:53 + 1 more... 10m kube-system kube-scheduler <none> 10m

with current parameters:
./kind create cluster --ipv6 --config config-ipv6.yaml

and config-ipv6.yaml

# this config file contains all config fields with comments kind: Config apiVersion: kind.sigs.k8s.io/v1alpha2 # 3 control plane node and 3 workers nodes: # the control plane node config - role: control-plane replicas: 3 # patch the generated kubeadm config with some extra settings kubeadmConfigPatches: - | apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration metadata: name: config networking: serviceSubnet: "fd00:1234::/112" # the three workers - role: worker # replicas specifes the number of nodes to create with this configuration replicas: 2 - role: external-load-balancer

@BenTheElder I'd like to add some e2e scenarios, could you help me or guide me?
Should those tests be part of this PR?

@aojea
hm, given weave supposedly does not support ipv6, how are you testing this?

./kind create cluster --ipv6

we need to remove the --ipv6 flag from the CLI, should be config only.
the CLI is only for the basic functionality.

@BenTheElder I'd like to add some e2e scenarios, could you help me or guide me?
Should those tests be part of this PR?

let's get the support in kind first, and then we can think about e2e.
ideally things like ipv6 should be owned by sig-network, but i can think we might be able to add this in the sig-cluster-lifecycle dashboard too.

i will try your PR later today myself, hopefully.

@neolit123 seems that the cluster control plane is ipv6 but the data plane is ipv4, let me do more tests changing to the cni bridge plugin and come back to you

@aojea nice.

podSubnet: "fd00:100::/64" serviceSubnet: "fd00:1234::/112"

do we have to instruct the users to patch the config with IPv6 CIDRs if they want IPv6 support.

kind create cluster --ipv6

ideally should be a flag in the kind Config.

Where should be the best place to add the ipv6 flag?
What do you think to add it in the new networking section of the v1alpha3 API #340 ?

What do you think to add it in the new networking section of the v1alpha3 API

yes, makes sense.

and how about this?

do we have to instruct the users to patch the config with IPv6 CIDRs if they want IPv6 support?

@neolit123 that's a good question but I miss context here, but I think that maybe can be a topic to be discussed in kubeadm.
I could check that kubeadm sets a default servicesubnet if none is provided, should we do the same for IPv6? if that's the case we should add a flag to kubeadm then

I could check that kubeadm sets a default servicesubnet if none is provided, should we do the same for IPv6? if that's the case we should add a flag to kubeadm then

here are kubeadm defaults for these CIDRs:
https://github.com/kubernetes/kubernetes/blob/b66e332d3c19ce78e00dd7c904fa29e8f6784ba0/cmd/kubeadm/app/apis/kubeadm/v1beta1/defaults.go#L31-L34

it not very likely that we will add a flag or a field in the kubeadm config for this.
IPv6 is explicit on the kubeadm side and up to the administrator to decide what stack to use. so kind as a kubeadm wrapper might have to handle this explicitly (as your patch is doing here).

at least that's how i see it for now.

aojea · 2019-02-25T21:21:43Z

@BenTheElder it's failing to verify the docker image -var _imagesBaseDockerfile = but that header's file says

// Code generated by go-bindata.
// sources:
// ../../../../images/base/Dockerfile
// ../../../../images/base/clean-install
// ../../../../images/base/entrypoint/main.go
// DO NOT EDIT!

Do I need to recreate the file? how can I do it?

neolit123 · 2019-02-25T21:26:49Z

Do I need to recreate the file? how can I do it?

./hack/update-generated.sh ?

aojea · 2019-02-25T22:11:52Z

/hold until I test the haproxy scenario, also I want to test it changing the CNI plugin to verify that IPv6 works ok, please add more comments and suggestions

pkg/cluster/internal/create/kubeadm-join.go

BenTheElder · 2019-02-27T02:31:16Z

[intend to start doing some more tests around this ~tomorrow]

Thanks again for working on this, apologies for the review latency on my end. I've been looking deeper into the CNI options FWIW, intend to do some more testing with calico 👍

aojea · 2019-03-04T01:32:33Z

@BenTheElder after playing with different CNI plugins with unsatisfactory results I explored other options.
I've found that @leblancd had a k8s cluster working with IPv6 using static routes and the standard CNI bridge plugin, so I tried to add the logic to kind but I've discarded that because I've found it will be very hard to maintain and to scale, specially for having multiple clusters in the same host.
I ended creating a Poor Man CNI plugin that I called kindnet leveraging the standard CNI plugins, it's very simple, it has a daemon running in each node that polls the API to get the podCIDR assigned to each node and add a static route.
I have to rebase the code and still need to do more testing but I wanted to start to clean up things and get the code ready to merge, can you let me know your opinion about kindnet?
What will be the best option to select IPv6 or IPv4? can we add it as an option of the new API in the networking section?

aojea · 2019-03-04T12:21:01Z

jobs are failing because of

W0304 12:03:42.352] Error: failed to create cluster: failed to generate kubeadm config content: failed to find an object with kubeadm.k8s.io_v1alpha3_ClusterConfiguration|config to apply the patch

it seems related to this #126 🤔

are the CI jobs using a different kubeadm version?

aojea · 2019-06-10T13:55:07Z