Kubeadm fails to bring up a HA cluster due to EOF error when uploading configmap #1321
Comments
|
@iverberk Is there a reason for adding the following extra args:
Could you try with this setting? |
|
No, that is a left-over from the 1.12 configuration. I will try to remove that and update the issue. |
|
sigh I guess sometimes you need someone else to tell you the obvious... that was the culprit, sorry for the hassle. This did work well in 1.12 and for some reason, that I can't remember anymore, this was a necessary configuration parameter to make it work. |
|
I thought this was solved but the problem still remains, even with the settings removed. Sometimes it works though. I'm not sure what the exact reason is, but most likely some kind of race condition. I've created a test repository to isolate the problem. In this repo: https://github.com/iverberk/kubeadm-cp-test you can find a test setup with Vagrant and Docker that show the problem when joining the second controller to the first controller. Hopefully this will illustrate the problem and make way for a solution. |
neolit123
added
help wanted
|
Ok, new information: bootstrapping is successful if I pre-pull the hyperkube image...I tested this with my own Ansible environment but will update the test repo as well to see if it works. |
|
I can't quite reproduce the same result in the test repo. I stumbled upon this because after the first installation I would reset the kubeadm installation and run it again. The second time it would succeed. I guess one of the differences is that the image is already there at that point. It is still a weird issue and I'd like to know how you test the bootrstrapping scenario and why you never see this behaviour. If this is a Vagrant thing we should be able to pinpoint it. |
|
@iverberk could you kindly retest after kubernetes/kubernetes#72030 merged? |
|
I'm hitting the same issue regardless of pre-pulling the images. The first master bootstraps successfully whereas the second master fails with Apparently, the second master isn't a master, but the third master has no trouble. The images were pre-pulled in all machines, so I guess pre-pulling doesn't affect anything. My apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
networking:
podSubnet: 10.244.0.0/16
apiServer:
certSANs:
- "PUBLIC_IP"
controlPlaneEndpoint: "PUBLIC_IP:PORT"@iverberk Could you confirm this by adding another master after the second one fails? |
|
I can confirm that the pre-pulling doesn't always work. It seemed to work reliably but the last time I got the same error. This remains some kind of timing issue unfortunately.
… Op 6 jan. 2019 om 21:00 heeft Ravi Shankar ***@***.***> het volgende geschreven:
I'm hitting the same issue regardless of pre-pulling the images. The first master bootstraps successfully whereas the second master fails with unexpected EOF when uploading some configuration. The weirdest part is that when I try adding another master node, it bootstraps successfully!
***@***.***:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-master-1 Ready master 28m v1.13.1
kube-master-2 Ready <none> 18m v1.13.1
kube-master-3 Ready master 86s v1.13.1
kube-worker-1 Ready <none> 26m v1.13.1
kube-worker-2 Ready <none> 25m v1.13.1
kube-worker-3 Ready <none> 25m v1.13.1
Here, the images were pre-pulled in all machines, so I guess pre-pulling doesn't . Apparently, the second master isn't a master, but the third master has no trouble.
@iverberk Could you confirm this by adding another master after the second one fails?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
timothysc
added
kind/bug
|
I have exact same problem. |
|
@iverberk @fmehrdad @wafflespeanut Could you kindly repeated the test against on one of the above version? |
|
I just retested with the repo that I created to reproduce this issue. This is the result of adding the second controller: So unless my repo is not following the correct procedure, or the version of Kubadm that is used (1.13.2) is not correct, this is still not fixed. @fabriziopandini would it be possible for you to evaluate my repo and make an assessment if it is correct and representative of a vanilla kubeadm HA bootstrap flow? |
|
@ereslibre could you kindly check again the EOF problem on your vagrant setup? |
I will have a look at it tonight, the race condition fixed by kubernetes/kubernetes#72030 was slightly different, I never saw this one, but I'm happy to try to reproduce the issue and try to find the root cause. I will report back. |
|
I have experimented the same behaviour that you guys. The weirdest thing is to find that the first join (master-2) fails, but the second one is successful (master-3). master-2 joining process yields: |
|
With https://github.com/iverberk/kubeadm-cp-test it's 100% reproducible. I didn't hit it with my project https://github.com/ereslibre/kubernetes-cluster-vagrant though. I am still in the process of checking what's wrong, but some observations: upon joining the second master, some processes on the first master crash: namely, the controller-manager, the scheduler and etcd afterwards. Scheduler: Controller manager: etcd keeps restarting on the first master too. etcd on the second master keeps failing to start: @masantiago are you experiencing similar errors in your setup? |
|
Yes, indeed. In fact, I ended up with the final status after several restarts of the Weave POD. It seems to be an unstable situation, confirmed when I shutdown the master-3. I am not able to access the cluster since then. It always responds with an unexpected EOF Do not hesitate to ask me any trace you require. I will have a look to your project @ereslibre. Did you also try with 13.x version? |
Kubernetes-cluster-vagrant is merely a project to be able to code easier on Kubernetes itself (soon will be deprecated completely by kind). I didn't focus on deploying existing released versions, but it shouldn't be hard to extend to support that usecase. @iverberk I am changing the code of your project a bit to confirm some things: iverberk/kubeadm-cp-test@master...ereslibre:master The main change I did is to avoid copying too many certificates and keys, because some certificates won't be valid if you copy all of them directly (certificate SANS use the detected IP address on each machine and some certificates just cannot be reused on all machines). After this change I no longer see crashes when growing etcd. So, what I can see at this point (with the changes applied) is that etcd takes a bit longer to start on the second controller, and since we are automatically using the stacked etcd cluster, the phase I think I'm not getting this problem on my project because in my case I create a base box with all the dependencies pulled, used for all machines. This means that I don't wait to pull for images on kubernetes-cluster-vagrant. I will report back when I have more information. |
|
I think my theory stands. So, what I did was to use: iverberk/kubeadm-cp-test@master...ereslibre:master. It's very important to only copy the certificates that need copying, and not the rest, because if certain certificates exist they won't be generated and their SANs won't include the proper IP addresses on the new machines. With the changes I previously linked I ran: And the node joined just fine, without any timeout. This is because I added the |
|
ok hopefully this can fixed then with #1341 |
|
So, two things after discussing with @fabriziopandini:
As for 1., despite we name the explicit certificates that need copying on the documentation (https://kubernetes.io/docs/setup/independent/high-availability/#steps-for-the-rest-of-the-control-plane-nodes), I think we can expect more people to try to copy I see this as a temporary solution until we have completely addressed the automatic transmission of secrets between masters when creating an HA cluster with kubeadm. Until we have a proper solution this would make it easier to copy things around between machines. |
|
@ereslibre thanks for the wrap up
What about adding an explicit warning on the document: don't copy all the certs!
Unfortunately automatic transmission of certs will be optional so, this fix should is necessary for v1.14 too; nevertheless, let's check if this can be fixed with a small change/eligible for backport in v1.13 .
Might be better error out with a better error message (instead of silently changing certs). Wdyt? |
I would also agree with this solution and feel it's a better one, because by automatically fixing the certificates we would be "promoting" the bad habit of copying everything when not everything is needed, so I'm good with erroring out and panicking the join in that case. We don't try to be smart, we just check if a certificate exists and if it doesn't match what we expect we abort. |
|
FYI I am only copying the files listed in https://kubernetes.io/docs/setup/independent/high-availability/ and still have this problem. |
|
I have been pre pulling the images using "kubeadm config images pull" ahead
of time. did not help.
I also tried 1.13.2 with no success.
…On Mon, Jan 14, 2019 at 9:59 AM Rafael Fernández López < ***@***.***> wrote:
@fmehrdad <https://github.com/fmehrdad> Can you double check if
prepulling the images in the new node before calling to kubeadm join
helps? If that's the case kubernetes/kubernetes#72870
<kubernetes/kubernetes#72870> is the issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1321 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AC2rtDs82V_j0uQnd5YO2rp7UV8foiHSks5vDMWegaJpZM4ZSaP7>
.
|
I need more information in order to know what's going on in your setup @fmehrdad. This issue reported by @iverberk comes with a repository that shows two different issues.
Both problems cause the same visible error, but the issues have a different nature. Can you please paste the configurations you are using to deploy the cluster and how your setup is done (HA...)? |
|
I've just tested with pre-pulling in second master, and the same behavior remains like @fmehrdad. My configuration is like that:
Vagrant like kubeadm-config.yaml where 172.168.33.20 is the VIP for the three masters using a keepalived and nginx load balancing. master1 Copying exactly the certs indicated in https://kubernetes.io/docs/setup/independent/high-availability/#stacked-control-plane-and-etcd-nodes master2 master3 Result: |
|
I can see something that will be probably related. When joining the second master to the cluster the Master 2 (~14 seconds): Master 3 (~40 milliseconds): So I can confirm that the first control plane join takes a bit longer, so it seems that we have yet another cause for this issue and this would match with @masantiago and @fmehrdad descriptions. I did the |
|
My problem was not related to k8s. My nginx-lb config had a very short timeout. I changed my proxy_timeout from 3s to 24h. Here is my config error_log /var/log/nginx/error.log warn; events { http { } stream { } |
|
/assign |
|
@ereslibre: GitHub didn't allow me to assign the following users: ereslibre. Note that only kubernetes members and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@ereslibre will be working on this one. /lifecycle active |
k8s-ci-robot
added
the
lifecycle/active
|
@fmehrdad Yesterday, I made a similar test but removing the LB (nginx) at all. I got to put all three masters in active, but:
Can you check if it is your case? |
|
Just tested @fmehrdad with your nginx config and I got the unexpected behaviour when shutting down masters. The replicas of PODs are not rescheduled and, moreover, when only the master 1 is left, I retrieve: It seems that the HA of either etcd or control plane is not working properly. I really appreciate your feedback in such case. |
If you have grown your cluster to 3 masters Please, let's keep this issue from now on for the certificate issue, since the reporter provided a repository that contained this problem and we are keeping this issue open because of that. This issue is split in 3 different cases:
For any related issue please refer to the explicit issues linked above, otherwise please let's open a new bug report since this one is already mixing very different things. Thank you! |
|
@ereslibre and others, just wanted to say a big thank you for investigating this! It's great to see such commitment on making kubeadm a great tool to use. |
|
Hello, Using k8s v1.12.1 in HA mode.
ACTIVE_MASTER_IP_ADDR -> is the current master node address where the keepalived is currently in master state (the other 2 are in slave/backup mode). HOW to reproduce: |
What keywords did you search in kubeadm issues before filing this one?
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version : kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:02:01Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Environment:
uname -a):Linux controller-1 4.15.0-38-generic kubeadm output should be clear and beautiful #41-Ubuntu SMP Wed Oct 10 10:59:38 UTC 2018 x86_64 x86_64 x86_64 GNU/LinuxWhat happened?
I'm trying to set up a HA cluster with three control plane nodes. I can successfully bootstrap the first controller but when I try to join the second controller it fails. After writing the etcd pod manifest it tries to write the new kubeadm-config (I guess with the updated controller api endpoints) but it fails with:
I'm using a haproxy loadbalancer in front of the three (to-be) API server nodes. HAproxy is quering the health endpoint of the API server and getting successful responses. Before joining the second controller I can successfully curl the endpoint with:
When the second controller joins the above curl will fail with an EOF and only start working about 20-30 seconds later. In the meantime the join command tries to upload the new config and crashes.
What you expected to happen?
I would have expected that the config uploading was successful, either by waiting for a healthy control plane or no problems with the API server in the first place.
How to reproduce it (as minimally and precisely as possible)?
I'm setting this up in a Vagrant environment but I guess it's no different than described on the https://kubernetes.io/docs/setup/independent/high-availability/ page. Here is my kubeadm config for the first controller:
Anything else we need to know?
These are some of the api server logs when the etcd join happens:
This is my haproxy config:
The text was updated successfully, but these errors were encountered: