-
Notifications
You must be signed in to change notification settings - Fork 715
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for using kubeadm without default route #3102
Comments
To follow up on #3075, there does indeed seem to be a bug within kubeadm even with a default route present:
It seems not only does kubeadm need a default route, but also seem to need an IPv4 default route, which is not present in IPv6 only networks anyway. I verified this on a second node where the behaviour is identical with default route present:
|
And on both nodes it works with the same workaround as before:
Same on the other node:
|
i don't think we want to update the documentation or modify kubeadm, because kubeadm is aligned with the IP detection mechanism of all k8s components. they all use default route. also, @uablrek @aojea do you remember that kubernetes/kubernetes ticket where we discussed that k8s does need a default route for core features. was it related to Services?
@telmich this is already supported by passing IPs to everything. this page has a note that is clear about that
|
You can and must do this, if you are setting your own routing rules, it means you are a power user and you are in control of the network, so you must do the corresponding IP planning and assignment Maybe the question is, how can I pass the IPs to the kubernetes components with kubeadm? |
Yes. Ref kubernetes/kubernetes#123120 |
there was this blog post on how to configure all components with kubeadm, but it did not get to a good state to be merged:
here it is: (EDIT: was just posted above) |
@telmich Check if the bogus default route mentioned in kubernetes/kubernetes#123120 (comment) can be used as a work-around.
|
Can you elaborate on this a little bit? I don't understand how any component of k8s actually requires a default route, because de-facto we have many k8s clusters running, healthy without a default route. We did not do any kind of tuning so far and all k8s components just work, the only exception is so far with kubeadm. |
I disagree with that as kubeadm does not support something like --address or --bind-address. The only, rather awkward way to make it work without a default route is by passing in above defined kubeadm-init-only.yaml. |
There is a difference between routing (where to go) and addressing (which source address to use). The routes on all systems work as well as do the addresses. Using ssh, curl, etc. all of these tools work on the machine. With one exception - that is why I created this feature request.
Maybe you can help me on this one: I actually don't understand why kubeadm is failing in the first place. I can curl the kube-apiserver, I can reach all kube components with curl without setting any kind of parameters. I honestly don't understand why kubeadm fails to connect in the first place. From an OS point of view, everything is working normally. |
how are you configurating bind addresses for them?
all k8s components have the same IP detection mechanism. it is summarized in this note:
|
Thanks a lot for the information! I've read that issue and while adding the default route seems to fix the particular problem, nothing I find in the ticket actually states that something requires a default route, besides then the link to the kubeadm documentation. From a network and OS perspective I would also claim that a default route is generally speaking unnecessary. What is required is connectivity between the nodes. Even the described kube-proxy issue might just be an implementation issue, which might not even be there with a CNI such as calico that can do bgp peering. What I really want to say with this is, I don't think a default route is actually necessary for kubeadm nor any of the k8s components. There might currently be dependencies on it in one or the other implementation way, but technically, networking wise, it is not needed. |
the way to configure them all is by using the kubeadm configuration file. check the linked blog post PR. EDIT: or this page: the IP detection that kubeadm does is the same as the kube-apiserver. if you pass the advertiseaddress field this detection is skipped as kubeadm thinks you know the address that you want. |
Address binding is not depending on routing. I think what you mean is "how to decide on which address to bind" and the default answer to that is: That's the default behaviour of virtually any network server. Maybe also to clarify, there might be 2 issues here, not sure how mixed we talk about them:
In both cases, this is usually best left to the operating system and should not be chosen by the application, unless
Because there are cases when a host has multiple addresses and the software needs to select a specific one, not left to the kernel.
Interesting, that is a bit the opposite case, having multiple default routes. From what I recall coding in C (maybe this is diffferent in golang?), NOT bind()ing to an IP address, but just using connect() does the expected thing: https://stackoverflow.com/questions/15673846/how-to-give-to-a-client-specific-ip-address-in-c Is there a limitation / requirement in go that forces kubeadm to bind in the first place? |
I just checked the generated manifest that kubeadm created and it contains:
And the initial configuration passed to kubeadm did in fact contain:
... which seems to somewhat explain why passing in the Initconfiguration makes kubeadm work on non-default route enabled systems (?), because it uses the InitConfiguration to set its source address? Sorry, slightly puzzled on this one. |
i'm assuming these are non-kubeadm clusters, so you must be configuring components in them somehow. if you want to use kubeadm and skip this detection just pass the config with advertiseAddress. |
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
comment is a bit misleading, but this is where the autodetection as per the kubeadm network docs comes into play. |
@telmich there are a lot of things under the hood that are not easy to infer. |
closing as documentation covers the status quo. |
FYI, this is CNI-plugin dependent (so "any" in the header is wrong). Installing without default route on some CNI-plugins:
In all cases a warning is printed:
And BTW, the bogus default route (#3102 (comment)) doesn't work. |
Optionally a default route to dev lo (doesn't work), to test kubernetes/kubeadm#3102. And a well needed cmd_env cleanup
What keywords did you search in kubeadm issues before filing this one?
I've checked previous tickets, specfically #3075 in which this issue was last discussed.
FEATURE REQUEST
Request
Add support for kubeadm to work without default route.
Background
I am aware of https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#network-setup saying we need a default route and I understand that there might be regressions running k8s without one.
However, we are running 30+ k8s clusters without default routes, because these k8s clusters are providing routing services to various infrastructures. Usually these are very small clusters, consisting of 1-4 nodes each, providing BGP, firewalling, NAT64 towards the infrastructure.
Technical considerations
One of the main reasons kubeadm currently fails is due to the selection of the IP address. From a network perspective, I see two very easy solutions to this:
In my opinion, (a) should be the default and (b) is already supported for many tools such as ping (using -I), ssh (using -B),
Follow up tasks
I would update the documentation above to reflect that newer versions of kubeadm can work without a default route, but that having a default route is recommended for most cases.
Versions
kubeadm version (use
kubeadm version
): kubeadm version: &version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.2", GitCommit:"39683505b630ff2121012f3c5b16215a1449d5ed", GitTreeState:"archive", BuildDate:"2024-07-03T09:11:54Z", GoVersion:"go1.22.5", Compiler:"gc", Platform:"linux/amd64"}(however seems to apply to all kubeadm versions)
Environment:
kubectl version
): anyuname -a
): anyWhat happened?
Trying to print the join command or running
kubeadm upgrade apply ...
fails on hosts without a default route.What you expected to happen?
I expect it to work, as the hosts in question have the full internet routing table:
They can reach any host, they just don't have a default route.
How to reproduce it (as minimally and precisely as possible)?
Create a node, remove the default route, keep required routes for pulling images.
Anything else we need to know?
Kubernetes provides a very good framework including managing network components. In many cases pods running on routers are actually running on the HostNetwork, which admittedly is not the default case, is a very useful use case.
We are running data centers all around the world and have been running routing services inside kubernetes now for almost 2 years. Upgrading kubeadm clusters without default route is a bit more dangerous and complicated then other systems because of the default route requirement.
So a typical upgrade or joining other nodes flow is at the moment:
So using kubeadm with a default route is possible, but dangerous and more complex, because routes are important and we don't want incorrect packets to be delivered to the wrong router (which we need to specific using the default route)
If there is any work needed in regards to selecting the right IP address or reasoning about the logic, I can be helping there. I am just not familiar with go/the kubeadm codebase.
The text was updated successfully, but these errors were encountered: