IPv6 and Klipper-LB #890
Comments
|
@bcr-ch Most probably we do not configure servicelb (ex-klipperlb) to work with IPv6. The main repo is here https://github.com/k3s-io/klipper-lb, and docs here https://docs.k3s.io/networking#service-load-balancer. Please don't hesitate to submit a PR, and please have a look at the debug section in the Readme, it provides useful details. |
mysticaltech
added
help wanted
|
Thanks for the response / looking into it. I think trying to address it and submitting a PR is outside my skill level. Nobody wants me writing code ;) I have done some investigation and it looks to me like during the initial K3S install / configuration we need to enable dual-stack support. Probably by passing the appropriate values to --cluster-cidr and --service-cidr. I found these links useful: This is where I kinda stall out, not sure which values are appropriate to pass, but I assume it could be specified in the kube.tf inside the k3s_exec_server_args variable. Hope that helps for anyone else who might read this and can give me a few more pointers. As always, great project and really cool work. |
|
@bcr-ch That's a good start, now if you can really pin point why the connection is refused, it would be great, you can do so by looking at the k3s logs, also making sure your DNS points to the AAAA IPv6 address(es) (you can have many of them, if you have many nodes)! |
|
Also, after verifying your DNS is configured correctly, please also post the logs here of Same for |
|
I have a feeling it might be a DNS issue, because now I remember, IPv6 traffic is automatically routed to IPv4 at the node level. |
|
@mysticaltech If I SSH to my node using this AAAA address it works (so it is accepting inbound connections on IPv6. I next try a telnet to the host for a simple whoami workload. Here you see it tries each of the IPv6 addresses before falling back to IPv4. Output of k logs svclb.... Output of ip a show When I try to deploy some test services to verify IPv6 support I get an error message as per below saying IPv6 is not configured on my cluster. A tcpdump of my telnet o the domain shows a RST right after the SYN. My hypothesis is that because we don't pass the --cluster-cidr and --service-cidr to the k3s binary as an arg during install we are not configuring the dual stack for the klipper lb. Therefore the NF_TABLES isn't setting up the required NAT entries for IPv6. |
|
@bcr-ch Thanks for the info, interesting. Yes the cluster cidr is passed, we use the /etc/rancher/k3s/comfig.yaml for that. How do you know telnet is not trying to use IPv4, maybe try with a dummy server work load, so you can test with curl, and double check that the http port is open on the hetzner firewall. |
|
Also, how did you make it so that your nodes do not have IPv4s, do you use a custom implementation? If so please post a dummy PR so we can see the diff 🙏 |
|
Hello, no my nodes all have both ipv4 and ipv6 addresses. For each node in my cluster I have an A record and a AAAA record all point to the same domain. For example (app.K3S.my.domain) If you run an nslookup for my domain it returns all records (the ipv4 and the ipv6). Curl behaves exactly the same as telnet. My client systems (macos or OpenSuse) prefer ipv6 so try those records first. When it fails it falls back to ipv4. |
|
@bcr-ch I ran this who issue through our good friend GPT-4 and here's what it says: ChatGPT:This is quite a detailed problem and your user has been very thorough in the diagnosis. Based on the information provided, here's an analysis and some recommended troubleshooting steps: Analysis:
Recommendations:
To summarize, the core issue seems to be the lack of explicit IPv6 support/configuration in You:With servicelb, I want to be able to listen to incoming IPv6 traffic. ChatGPT:If you want
For example: apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: MyApp
ports:
- protocol: TCP
port: 80
targetPort: 9376
type: LoadBalancer
ipFamilyPolicy: PreferDualStack
Finally, once everything is set up, test the load balancer externally using IPv6 to ensure it's routing traffic as expected to your services. You:Do I need dual stack? For that? ChatGPT:No, you do not necessarily need dual stack (support for both IPv4 and IPv6 simultaneously) to have If you only care about IPv6 traffic and your nodes already have IPv6 access, you can set up your Kubernetes service to only use IPv6. However, without dual-stack, the service will not be reachable via IPv4. Here's what you'd do:
For example: apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: MyApp
ports:
- protocol: TCP
port: 80
targetPort: 9376
type: LoadBalancer
ipFamily: IPv6
Remember, without dual-stack, the services set to IPv6 will not be reachable via IPv4. If that's acceptable for your use case, then you don't need to worry about setting up dual-stack. However, if you want the flexibility of reaching services via both IP versions, then dual-stack becomes necessary. You:What it net.ipv6.conf.all.accept_ra=2, do I need it? ChatGPT:The The values for
In a typical client or host setting, forwarding is disabled, and the machine will process RAs by default. However, on routers or any machine with IP forwarding enabled (i.e., In certain situations, especially with some more complex Kubernetes setups where nodes might act as routers but still need to autoconfigure addresses using RAs, the value of If you're setting up Kubernetes on nodes that have forwarding enabled but still need to receive and process IPv6 router advertisements for address autoconfiguration or other information, then setting |
|
@bcr-ch So what I recommend is trying with Hetzner LB first, if it works there with IPv6, then the problem is not with kubernetes but with Klipper-LB/ServiceLB or the the way IPv6 behaves on the nodes. And then we can try tweaking the IPv6 config for the nodes. But first important to confirm that. |
|
Thanks for the GPT advice ;) And of course your support as well. So I modified my kube.tf and disabled klipper deploying instead a Hetzner LB. I now how a LB (with both an IPv6 and an IPv4) address redirecting to my three agent nodes. I have updated my DNS so I have an A record and AAAA record pointing to the LB. Now when I access my website "example.my.domain" it first tries the IPv6 address and succeeds. So I'm getting the feeling that the problem sits with how IPv6 on the nodes is configured with ServiceLB. |
|
@bcr-ch Exactly, thanks for confirming this! I will have a look. |
|
@bcr-ch Please try adding this to the cloud-init template: |
|
I cannot easliy test on my end because I do not have IPv6 locally. So try setting the above, if it work, we will add that step to the project. |
|
I modified my locals.tf file to modify the cloud-init settings as suggested above and redeployed my cluster. Unfortunately I get the same behaviour as before. I have confirmed by connecting to my Agent Node that the setting applied
produces the output: I will play with the initial K3S server install as I think there may be something in there I can tweak. Will post back on my success or failure ;) |
|
Ok, so more digging in my test environment and I've now come across this: /var/lib/rancher/k3s/agent/etc/flannel/net-conf.json Based on my reading of this old issue in flannel So our IPv6 is not getting enabled in flannel, which is used by k3s by default. |
|
@bcr-ch Thanks for trying the ipv6-access value. About the above, I believe that is for dual stack and probably not what we want as the IPv6 traffic is coming from outside. @kube-hetzner/core Any ideas on this please? I'm dry on ideas. |
|
Yes you are correct... it doesn't work because we are using the private network for the internal traffic. This network doesn't support IPv6 so getting it to listen on IPv6 wouldn't work. Also not what we want. We don't want a dual stack cluster, just want the svclb front-end to accept IPv6 traffic. I found this old post for klipper-lb. I installed socat onto the cluster, and ran it as follows: Now when I telnet to my cluster the IPv6 address works ;). It accepts the traffic and redirects it to the svclb listening on IPv4. This sounds like a feature missing in svclb @mysticaltech, I suppose we can close the request if you agree. Sounds very much like a a limitation in klipper-lb. I can work on a PR where I have an option that would install SOCAT and create/start services on each node to redirect traffic... but TBH it sounds like a hack and probably not something you want. Also I don't do a lot of dev anymore so it will take me a while :) Thanks for all the support |
|
Very good find @bcr-ch 🍾 ! This was good team work, well done. A PR would have be great but since we do not know exactly which port needs connecting to IPv4, it's not practical. And anyways, thank goodness the Hetzner LB automatically takes care of that translation. |
|
Now if there is a way for socat to forward all IPv6 ports to their equivalent IPv4, then why not. If you're up to it, PR is most welcome, and should be quite quick. |
|
@bcr-ch Just saw the donation, thank you for that, really appreciate it! 🙏 |
Description
I am using klipper-lb and traefik instead of the Hetzner Load Balancer.
When i use ipv4 address of my agent nodes it works no problem. When I use ipv6 address I get 'Connection refused'
Seems that the host is rejecting inbound IPv6. I have the default FW rules and no issue
Kube.tf file
Screenshots
No response
Platform
Linux
The text was updated successfully, but these errors were encountered: