IPv6 support

[jpds]

IPv6 should be moving to beta in 1.10 and is in alpha in 1.9 - does kube-router have any integration with the protocol version?

[murali-reddy]

@jpds honestly not much thought went in to supporting IPv6 at this point. But looks like at some point kube-router should start supporting it. If there is need from multiple users then cetainly support for IPv6 can be prioiritzed.

[geosword]

+1

[uablrek]

Please consider it. Ipvs, gobgb and ip6tables are supporting ipv6. I think the effort is smaller than one think since the commands are basically the same but with other addresses (don't know about the netlink though).

[murali-reddy]

https://github.com/vishvananda/netlink libray that is used by kube-router supports IPV6 as well.

[telmich]

+1 - looking forward very much to ipv6 support in kube-router (we are on ipv6 only networks)

[murali-reddy]

There is backlog of issue currently for 0.2 release, We can work on this work 0.3 release.

Meanwhile I am happy to accept any partial fixes towards IPv6 support.

[telmich]

What is exactly missing / needed in kube-router to fully support IPv6?

[pmichali]

As a follow-up to issue #470 I've changed the address family temporarily and hit a few more issues/concerns/unexpected items, and figured I'd document them here FYI.

First, I started with an existing cluster, that had bridge CNI plugin. The config called out a /72 pod subnet CIDR and the CNI configurations on nodes defined /72 subnets. When I started up kube-router, it created a new CNI config, that created a new bridge and /88 subnet, on each node. Should I have created the cluster w/o specifying the bridge CNI plugin (and static routes for pod networks to communicate with each node)?

Second, it looks like the SyncNodeIPSets() has ipset call create with this /88 info and by default, ipset is doing IPv4. Looks like "family inet6" needs to be specified (thanks @murali-reddy). Here is the error:

E0620 11:42:34.532817       1 network_routes_controller.go:110] Failed initial ipset setup: Failed to sync Pod Subnets ipset: ipset v6.34: Syntax error: '88' is out of range 0-32
E0620 11:42:34.551302       1 network_routes_controller.go:129] Failed to enable required policy based routing: Failed to add ip rule due to: exit status 2

Third, there is another error in kube-router, which combined with the previous error causes it to fail to start. The message is:

I0620 11:42:34.557856       1 network_routes_controller.go:194] Starting network route controller
E0620 11:42:34.705180       1 network_routes_controller.go:200] Failed to start node BGP server: Failed to start BGP server due to : listen tcp6 [::]:179: bind: address already in use

The netstat output shows some listening (on this host):

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 fd00:20::2:179          :::*                    LISTEN      -

[pmichali]

On the first issue, in syncNodeIPSets() I tried ensuring that the Create() call would specify that the IP family is IPv6. However, I found that the Create() method was not called (there is already a psSet and it was using ipv4. It then calls Refresh() with the IPv6 cluster CIDRs for the pod ets on each node, and complains of the incompatibility.

Not sure where the initial sets are created, so that they can be made IPv6.

P.S. I'll be on PTO for a week or so, but will resume investigation, when I get back.

[pmichali]

Here are some additional things that I see with the current kube-router code...

• The docker image has iptables, but not ip6tables
• In controllers/routing/pbr.go it uses ip commands, but is not using the "-6" argument for IPv6.
• In controllers/routing/pod_egress.go it creates an iptables object, instead of an ip6tables object

I've done some hacks to the code to force the above to use IPv6 equivalents.

The current issue I have is that the GBP server does not start. For IPv4 mode, it seems like the local address list passed to the bgp server that is started has [ 10.192.0.2, ::], for IPv4 and IPv6. I see listening on the V4 address port 179, and listening on V6 any address for port 179 and two other ports (I have 3 nodes).

I'm unsure how to set the local address list, when running in IPv6 (only) mode. I've done some guessing, without any success [fd00:20::2, fd00:20::2], [::, ::], [::1, ::1], [fd00:20::2, ::], [::, fd00:20::2], each saying port 179 is in use. If I try to use [0.0.0.0, fd00:20::2], I get a panic saying that the slice bounds is out o range.

Can anyone explain how the local address list should be configured for use in IPv6 mode? Do I need to force an IPv4 address on the interface (it is IPv6 only right now)?

[pmichali]

I have not been able to figure out how to pass the right arguments for the local listen addresses to the go routine that starts goBGP, when operating in IPv6 mode.

[murali-reddy]

@pmichali ok let me give it a try.

GoBGP folks also has a slack, https://slackin-gobgp.mybluemix.net they are pretty responsive.

[murali-reddy]

stay tuned. thanks to @uablrek IPv6 is taking good shape

https://github.com/cloudnativelabs/kube-router/blob/master/docs/ipv6.md

[aauren]

Bumping this up since IPv6 is now fully supported as of Kubernetes 1.16. We'll probably want to revisit this soon.

[rearden-steel]

Is it possible now to use ipv6 as an externalIP with kube-router? @murali-reddy

[murali-reddy]

@rearden-steel I belive no. Service proxy part is not yet compatible with IPv6.

https://github.com/cloudnativelabs/kube-router/blob/master/docs/ipv6.md is the up to date status.

CC: @uablrek

[karibertils]

+1 looking forward to this one

[kvaps]

BTW I switched from Kube-router to Cilium.
IPv6 and dual stack is made really well there

[telmich]

@kvaps did you try ipv6 only in cilium?

[kvaps]

Well there is not much plugins which support IPv6, for now only Calico and Cilium, also Contiv.
Yes. I've tried only cilium.

[Ulexus]

DANM also supports IPv6 very well. I hadn't had much luck with Cilium until 1.8. GoBGP itself supports IPv6 well.

[telmich]

@kvaps Sorry, I did not mean to say "did you only test cilium", but I meant to say "did you test IPv6 only" in cilium :)

Because what is of interest for me is IPv6 only k8s clusters without any IPv4 being involved.

[kvaps]

I have tested dual stack, but IPv6-only should be even simpler.
You don't need to enable any future-gates in Kubernetes for that.
I'm sure, IPv6 will work out-of-box.

[telmich]

@kvaps Last time I checked all the components had hardcoded IPv4 addresses for discovery, which won't work in IPv6 only environments

[jnummelin]

Could someone summarize the state of ipv6 support? The docs says it's not supported, but those statements are quite old (~y2019). There's been quite a few ipv6 related PRs being merged in the past year or so hence I'm wondering to what extent it should be working already?

Any references how people are using/testing kube-router with ipv6?

[yuchunyun]

+1

[arianvp]

I tried it out and it breaks quite quickly:

I0201 13:26:28.832910       1 version.go:21] Running /usr/local/bin/kube-router version v1.4.0, built on 2022-01-05T17:01:42+0000, go1.17.5
I0201 13:26:28.936663       1 metrics_controller.go:164] Starting metrics controller
I0201 13:26:28.944673       1 network_routes_controller.go:1298] Could not find annotation `kube-router.io/bgp-local-addresses` on node object so BGP will listen on node IP: 2001:19f0:6c01:2e40:5400:3ff:fed4:8e66 address.
E0201 13:26:28.968309       1 network_routes_controller.go:179] Failed to enable required policy based routing: failed to add ip rule due to: exit status 2
I0201 13:26:28.973336       1 network_routes_controller.go:242] Setting MTU of kube-bridge interface to: 1480
I0201 13:26:28.974416       1 network_routes_controller.go:273] Starting network route controller
time="2022-02-01T13:26:28Z" level=warning msg="listen failed" Error="listen tcp: address 2001:19f0:6c01:2e40:5400:3ff:fed4:8e66:50051: too many colons in address" Key="2001:19f0:6c01:2e40:5400:3ff:fed4:8e66:50051" Topic=grpc
time="2022-02-01T13:26:28Z" level=fatal msg="failed to listen grpc port: listen tcp: address 2001:19f0:6c01:2e40:5400:3ff:fed4:8e66:50051: too many colons in address"

with the setup where each node peers with a ToR router:

  args:
        - --run-router=true
        - --run-firewall=false
        - --run-service-proxy=false
        - --bgp-graceful-restart=true
        - --cluster-asn=4288000094
        - --peer-router-asns=64515
        - --peer-router-ips=2001:19f0:ffff::1
        - --peer-router-passwords=xxxxxxx
        - --router-id=2001:19f0:ffff::1
        - --metrics-port=8080

[arianvp]

kube-router/pkg/controllers/routing/network_routes_controller.go

Line 989 in b9a9246

gobgp.GrpcListenAddress(nrc.nodeIP.String() + ":50051" + "," + "127.0.0.1:50051"))

seems to be what it's choking on.

when it's an ipv6 address the format should be fmt.Sprintf("[%s]:50051", nodeIP)

[arianvp]

I added a few more fixes: #1249

[telmich]

What is the current state of IPv6 in kube-router? Is it possible to try using it an IPv6 only environments?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[StevenACoffman]

#1386

[aauren]

Fixed by: #1386 & #1544