workcluster(one control plane + one node) upgrade #4493

chymy · 2021-04-20T01:44:13Z

I created a workcluster: one master + one node, upgrading the workcluster always fails. Does CAPI not support the upgrade for single master ?

Environment:

Cluster-api version: v0.3.11
Minikube/KIND version:
Kubernetes version: (use kubectl version):
OS (e.g. from /etc/os-release):

/kind bug
/help

The text was updated successfully, but these errors were encountered:

k8s-ci-robot · 2021-04-20T01:44:14Z

@chymy:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

I created a workcluster: one master + one node, upgrading the workcluster always fails. Does CAPI not support the upgrade for single master ?

Environment:

Cluster-api version: v0.3.11

Minikube/KIND version:

Kubernetes version: (use kubectl version):

OS (e.g. from /etc/os-release):

/kind bug
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sbueringer · 2021-04-20T04:02:15Z

@chymy It does. Our upgrade test uses 1 control plane node, a machine deployment with 3 replicas and a machinepool with 3 replicas. This is tested for:

1.18 => 1.19
1.19 => 1.20
1.20 => 1.21
1.21 => 1.22 (k/k main branch)

Test runs: https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-main-1-18-1-19
Test code: https://github.com/kubernetes-sigs/cluster-api/blob/master/test/e2e/cluster_upgrade.go#L87-L153

I think we'll need more details to find out why it fails in your case.

chymy · 2021-04-21T02:54:24Z

Thanks @sbueringer , It should be related to infrastructure provider.

chymy · 2021-04-23T03:16:34Z

Hi, @sbueringer,

cluster-api/controlplane/kubeadm/controllers/scale.go

Lines 118 to 142 in 46dc43a

    
           // If KCP should manage etcd, If etcd leadership is on machine that is about to be deleted, move it to the newest member available. 
        
           if controlPlane.IsEtcdManaged() { 
        
           	etcdLeaderCandidate := controlPlane.Machines.Newest() 
        
           	if err := workloadCluster.ForwardEtcdLeadership(ctx, machineToDelete, etcdLeaderCandidate); err != nil { 
        
           		logger.Error(err, "Failed to move leadership to candidate machine", "candidate", etcdLeaderCandidate.Name) 
        
           		return ctrl.Result{}, err 
        
           	} 
        
           	if err := workloadCluster.RemoveEtcdMemberForMachine(ctx, machineToDelete); err != nil { 
        
           		logger.Error(err, "Failed to remove etcd member for machine") 
        
           		return ctrl.Result{}, err 
        
           	} 
        
           } 
        
           if err := workloadCluster.RemoveMachineFromKubeadmConfigMap(ctx, machineToDelete); err != nil { 
        
           	logger.Error(err, "Failed to remove machine from kubeadm ConfigMap") 
        
           	return ctrl.Result{}, err 
        
           } 
        
           logger = logger.WithValues("machine", machineToDelete) 
        
           if err := r.Client.Delete(ctx, machineToDelete); err != nil && !apierrors.IsNotFound(err) { 
        
           	logger.Error(err, "Failed to delete control plane machine") 
        
           	r.recorder.Eventf(kcp, corev1.EventTypeWarning, "FailedScaleDown", 
        
           		"Failed to delete control plane Machine %s for cluster %s/%s control plane: %v", machineToDelete.Name, cluster.Namespace, cluster.Name, err) 
        
           	return ctrl.Result{}, err 
        
           }

I create a workcluster, apiServerEndpoint is loadbalancer. During the upgrade, if etcd member is deleted by calling workloadCluster.RemoveEtcdMemberForMachine in advance, the load balancer is likely to connect to the old controlplane node, which will cause workloadCluster.RemoveMachineFromKubeadmConfigMap to fail to execute all the time,the old controlplane node cannot be deleted. Why not put workloadCluster.RemoveMachineFromKubeadmConfigMap in front of controlPlane.IsEtcdManaged.

Thanks.

chymy · 2021-04-23T08:26:42Z

While removing etcd member, should also stopping apiserver

sbueringer · 2021-04-23T10:31:37Z

Let's reopen the issue for now to discuss this point. I cannot really say why the code is as it is right now, so let's see what others will say.

/reopen

k8s-ci-robot · 2021-04-23T10:31:42Z

@sbueringer: Reopened this issue.

In response to this:

Let's reopen the issue for now to discuss this point. I cannot really say why the code is as it is right now, so let's see what others will say.

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

chymy · 2021-04-25T01:21:58Z

After testing, before removing etcd member, remove the static yaml of apiserver first, and delete the controlplane node is successful. So just make sure that the apisever stops and doesn't start any more, the load balancer will not connect to old controlplane node.

vincepri · 2021-04-26T17:32:01Z

/retitle workcluster(one control plane + one node) upgrade

chymy · 2021-05-26T08:12:41Z

After testing, apiServerEndpoint is loadbalancer, adjust the following code order, upgrade and downgrade are successful.
because call the workloadCluster.RemoveMachineFromKubeadmConfigMap and r.Client.Delete before controlPlane.IsEtcdManaged() etcd , and the infrastructure is triggered to clean up successfully the old apisever instance used by loadbalancer. If RemoveEtcdMemberForMachine ahead of time, then the workloadCluster.RemoveMachineFromKubeadmConfigMap and r.Client.Delete are likely to fail.

cluster-api/controlplane/kubeadm/controllers/scale.go

Lines 118 to 142 in 46dc43a

    
           // If KCP should manage etcd, If etcd leadership is on machine that is about to be deleted, move it to the newest member available. 
        
           if controlPlane.IsEtcdManaged() { 
        
           	etcdLeaderCandidate := controlPlane.Machines.Newest() 
        
           	if err := workloadCluster.ForwardEtcdLeadership(ctx, machineToDelete, etcdLeaderCandidate); err != nil { 
        
           		logger.Error(err, "Failed to move leadership to candidate machine", "candidate", etcdLeaderCandidate.Name) 
        
           		return ctrl.Result{}, err 
        
           	} 
        
           	if err := workloadCluster.RemoveEtcdMemberForMachine(ctx, machineToDelete); err != nil { 
        
           		logger.Error(err, "Failed to remove etcd member for machine") 
        
           		return ctrl.Result{}, err 
        
           	} 
        
           } 
        
           if err := workloadCluster.RemoveMachineFromKubeadmConfigMap(ctx, machineToDelete); err != nil { 
        
           	logger.Error(err, "Failed to remove machine from kubeadm ConfigMap") 
        
           	return ctrl.Result{}, err 
        
           } 
        
           logger = logger.WithValues("machine", machineToDelete) 
        
           if err := r.Client.Delete(ctx, machineToDelete); err != nil && !apierrors.IsNotFound(err) { 
        
           	logger.Error(err, "Failed to delete control plane machine") 
        
           	r.recorder.Eventf(kcp, corev1.EventTypeWarning, "FailedScaleDown", 
        
           		"Failed to delete control plane Machine %s for cluster %s/%s control plane: %v", machineToDelete.Name, cluster.Namespace, cluster.Name, err) 
        
           	return ctrl.Result{}, err 
        
           }

vincepri · 2021-07-06T17:22:53Z

/assign @sbueringer
/milestone v0.4
/priority important-soon

A good opportunity here is also to write a single node control plane upgrade e2e test in our periodics

sbueringer · 2021-07-09T09:03:40Z

@vincepri Just reviewed the whole issue again. We already have e2e tests which are testing this, our regular upgrade tests. (#4493 (comment)). I don't think I've ever seen this issue in CI.

Short summary (from my point of view), our code does the following right now:

scaleDownControlPlane
- if etcd managed:
  - forward etcd leader ship to newest control plane node
  - remove etcd member
- remove machine from kubeadm config map
- delete machine

The problem seems to be that after removing the etcd member the API server on the same machine goes down and the loadbalancer still sends requests to this node even though the API server is down (which I guess can always happen until the machine is dropped from the lb pool). So I assume this maybe doesn't happen in CI because CAPD is relatively fast.

I wonder why this isn't just retried if RemoveMachineFromKubeadmConfigMap fails (i.e. if our code here is reentrant). It would be really helpful to get logs to analyze this behaviour based on CAPI v1alpha4.

We could also change the order, I think that's safe although I'm not entirely sure.

I think overall our code has to be robust enough to handle occasionally failed requests (and it usually is, because we just retry failed reconciles). So changing the order would just be hacking around the problem (imho).

@fabriziopandini @vincepri WDYT?

fabriziopandini · 2021-07-09T09:40:15Z

I don't see problem in changing the order (also kubeadm doesn't use the list of etcd endpoint in the config map anymore since 1.18 and it was removed in 1.22, see #4423)

However I agree we should understand where our code is not re-entrant.
@chymy would it be possible for you to try to reproduce while adding some more logs in KCP?

chymy · 2021-07-13T03:14:47Z

After removing the etcd member, the API server on the same machine goes down, but the API server status is running:

[root@cmcluster-control-plane-xx5t4 ubuntu]# kubectl get po -A
NAMESPACE     NAME                                                    READY   STATUS             RESTARTS   AGE
kube-system   calico-kube-controllers-8899c4d5-dsgtq                  1/1     Running            3          82s
kube-system   calico-node-m7b6b                                       0/1     Running            1          2m49s
kube-system   calico-node-r58rw                                       1/1     Running            0          11m
kube-system   calico-node-s5qzg                                       0/1     Running            2          3m3s
kube-system   coredns-54cfb99b45-68bnv                                1/1     Running            0          21m
kube-system   coredns-54cfb99b45-nv4bl                                1/1     Running            0          21m
kube-system   etcd-cmcluster-control-plane-k8xk9                      1/1     Running            0          2m46s
kube-system   etcd-cmcluster-control-plane-xx5t4                      0/1     CrashLoopBackOff   3          22m
kube-system   kube-apiserver-cmcluster-control-plane-k8xk9            1/1     Running            0          2m47s
kube-system   kube-apiserver-cmcluster-control-plane-xx5t4            1/1     Running            0          22m
kube-system   kube-controller-manager-cmcluster-control-plane-k8xk9   1/1     Running            0          2m48s
kube-system   kube-controller-manager-cmcluster-control-plane-xx5t4   1/1     Running            1          22m
kube-system   kube-proxy-ch9n5                                        0/1     ImagePullBackOff   0          2m49s
kube-system   kube-proxy-pfnbt                                        0/1     ImagePullBackOff   0          3m3s
kube-system   kube-proxy-txhlk                                        1/1     Running            0          21m
kube-system   kube-scheduler-cmcluster-control-plane-k8xk9            1/1     Running            0          2m48s
kube-system   kube-scheduler-cmcluster-control-plane-xx5t4            1/1     Running            1          22m

It can be seen from the above, the etcd member has been removed. the API server pod status is running on the same machine, the loadbalancer communicates with the apisever, it thinks apisever is good, the loadbalancer still sends requests to this node, but the etcd is CrashLoopBackOff , so calling the workloadCluster.RemoveMachineFromKubeadmConfigMap is very likely to fail.
the controlpalne logs:

I0713 02:58:43.597652       1 scale.go:205]  "msg"="Waiting for control plane to pass preflight checks" "cluster-name"="cmcluster" "name"="cmcluster-control-plane" "namespace"="default" "failures"="machine cmcluster-control-plane-pkd4x reports EtcdPodHealthy condition is false (Info, Waiting for startup or readiness probes)"
I0713 02:58:44.994125       1 controller.go:324] controller-runtime/manager/controller/kubeadmcontrolplane "msg"="Rolling out Control Plane machines" "cluster"="cmcluster" "name"="cmcluster-control-plane" "namespace"="default" "reconciler group"="controlplane.cluster.x-k8s.io" "reconciler kind"="KubeadmControlPlane" "needRollout"=["cmcluster-control-plane-zh4qr"]
I0713 02:58:45.522274       1 scale.go:205]  "msg"="Waiting for control plane to pass preflight checks" "cluster-name"="cmcluster" "name"="cmcluster-control-plane" "namespace"="default" "failures"="machine cmcluster-control-plane-pkd4x reports EtcdPodHealthy condition is false (Info, Waiting for startup or readiness probes)"
I0713 02:58:46.469894       1 controller.go:324] controller-runtime/manager/controller/kubeadmcontrolplane "msg"="Rolling out Control Plane machines" "cluster"="cmcluster" "name"="cmcluster-control-plane" "namespace"="default" "reconciler group"="controlplane.cluster.x-k8s.io" "reconciler kind"="KubeadmControlPlane" "needRollout"=["cmcluster-control-plane-zh4qr"]
E0713 02:58:49.143813       1 workload_cluster.go:329]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: rpc error: code = Unavailable desc = transport is closing"
E0713 02:59:08.846999       1 workload_cluster.go:329]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: Get \"https://192.116.1.53:2602/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"


E0713 02:59:27.302484       1 workload_cluster.go:329]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: Get \"https://192.116.1.53:2602/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
E0713 02:59:47.729497       1 workload_cluster.go:329]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: Get \"https://192.116.1.53:2602/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
E0713 03:00:14.349689       1 workload_cluster.go:329]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: Get \"https://192.116.1.53:2602/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
E0713 03:00:39.478106       1 workload_cluster.go:329]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: Get \"https://192.116.1.53:2602/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"


E0713 03:01:13.807685       1 workload_cluster.go:329]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: Get \"https://192.116.1.53:2602/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
E0713 03:01:50.711208       1 workload_cluster.go:329]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: Get \"https://192.116.1.53:2602/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"

E0713 03:02:43.043175       1 workload_cluster.go:329]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: Get \"https://192.116.1.53:2602/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
E0713 03:03:46.310975       1 workload_cluster.go:329]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: Get \"https://192.116.1.53:2602/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
E0713 03:03:46.311025       1 scale.go:132]  "msg"="Failed to remove machine from kubeadm ConfigMap" "error"="timed out waiting for the condition" "cluster-name"="cmcluster" "name"="cmcluster-control-plane" "namespace"="default"
E0713 03:03:56.490621       1 controller.go:302] controller-runtime/manager/controller/kubeadmcontrolplane "msg"="Reconciler error" "error"="timed out waiting for the condition" "name"="cmcluster-control-plane" "namespace"="default" "reconciler group"="controlplane.cluster.x-k8s.io" "reconciler kind"="KubeadmControlPlane"

I0713 03:04:05.113778       1 controller.go:324] controller-runtime/manager/controller/kubeadmcontrolplane "msg"="Rolling out Control Plane machines" "cluster"="cmcluster" "name"="cmcluster-control-plane" "namespace"="default" "reconciler group"="controlplane.cluster.x-k8s.io" "reconciler kind"="KubeadmControlPlane" "needRollout"=["cmcluster-control-plane-zh4qr"]
E0713 03:04:25.707031       1 controller.go:302] controller-runtime/manager/controller/kubeadmcontrolplane "msg"="Reconciler error" "error"="failed to update the kubernetes version in the kubeadm config map: error getting kube-system/kubeadm-config configmap from target cluster: Get \"https://192.116.1.53:2602/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" "name"="cmcluster-control-plane" "namespace"="default" "reconciler group"="controlplane.cluster.x-k8s.io" "reconciler kind"="KubeadmControlPlane"

I0713 03:04:34.186963       1 controller.go:324] controller-runtime/manager/controller/kubeadmcontrolplane "msg"="Rolling out Control Plane machines" "cluster"="cmcluster" "name"="cmcluster-control-plane" "namespace"="default" "reconciler group"="controlplane.cluster.x-k8s.io" "reconciler kind"="KubeadmControlPlane" "needRollout"=["cmcluster-control-plane-zh4qr"]

sbueringer · 2021-07-13T06:01:28Z

@chymy Looks like the retry works as expected. I would have expected a 50% success ratio with 2 control plane nodes. Which provider are you using?

I'm a bit confused:

API server on the same machine goes down
the API server pod status is running on the same machine, the loadbalancer communicates with the apisever, it thinks apisever is good

Is that because the API server is still up but all requests fail because it cannot communicate with the etcd on the same host?

chymy · 2021-07-13T06:22:49Z

@chymy Looks like the retry works as expected. I would have expected a 50% success ratio with 2 control plane nodes. Which provider are you using?

I'm a bit confused:

API server on the same machine goes down
the API server pod status is running on the same machine, the loadbalancer communicates with the apisever, it thinks apisever is good

Is that because the API server is still up but all requests fail because it cannot communicate with the etcd on the same host?

I use openstack.

Sorry. yes, the API server is still up(not down). LB and APIServer are long connections, it seems that it is feasible to retry the work

sbueringer · 2021-07-13T07:38:21Z

@fabriziopandini What do you think?

I would have assumed our retries would succeed eventually because we should hit the still working API server at some point, but that doesn't seem to be the case. Ideally the loadbalancer would either drop the not-working API server at some point because of failing health checks or we would sooner or later get the request through to the working API server.

As that's not the case it would be fine for me to change the order to mitigate the issue (as it's not a problem for us to change it).

chymy · 2021-07-13T07:51:41Z

In fact, by changing the order, LB can detect that the old apisever is no longer working, so it can connect to the new apisever

sbueringer · 2021-07-13T08:02:31Z

In fact, by changing the order, LB can detect that the old apisever is no longer working, so it can connect to the new apisever

Just out of curiosity (I don't really know how this stuff works): How does that work? I thought we only remove the server from the ConfigMap. How does that signal to the loadbalancer that the API server no longer works?

chymy · 2021-07-13T08:16:53Z

I think r.Client.Delete triggers the infrastructure provider to clean up the VM, and then the old apiserver instance will be cleaned up from LB.

fabriziopandini · 2021-07-13T12:54:56Z

TBH, I think that something should be tuned in the LB configuration, given that according to the log above it can't detect the API server failed in 4 minutes, and all the calls goes on a single endpoint (stickiness?). This is a problem no matter of the order of the tree operations we are discussing, because other API servers calls are going happen for sure immediately after.

However, as per #4493 (comment), I don't see particular problems in the proposed change, and if this help in making our code a little bit more robust for such use cases...

sbueringer · 2021-07-13T13:44:38Z

Sorry missed something before. I think it might be dangerous to change the order. New code would be:

scaleDownControlPlane
- remove machine from kubeadm config map
- delete machine
- if etcd managed:
  - forward etcd leader ship to newest control plane node
  - remove etcd member

This means (imho) that we have a race condition between deletion of the server and moving etcd leadership. I'm not sure if that's really a good idea. Forwarding etcd leader ship is probably really fast in most cases (not sure what might slow it down), but if the server deletion is faster this can probably lead to some really bad effects (?).

EDIT: obvious option I missed:

scaleDownControlPlane
- if etcd managed:
  - forward etcd leader ship to newest control plane node
- remove machine from kubeadm config map
- delete machine
- if etcd managed:
  - remove etcd member

But I'm not sure if that's a good idea either. It feels safer/cleaner to first remove the etcd member and then the machine/server. (when using kubeadm (in the usual case without CAPI) you also first remove the member via kubeadm reset and not just delete the machine/server)

sbueringer · 2021-07-19T07:28:19Z

@fabriziopandini what do you think, should we still change the order?

fabriziopandini · 2021-07-21T16:16:35Z

The only change of order that makes sense IMO is

scaleDownControlPlane
- remove machine from kubeadm config map
- if etcd managed:
  - forward etcd leader ship to newest control plane node
  - remove etcd member
- delete machine

Because delete machine should always be called after completing all the actions on the components hosted on that machine.

However the context and limitations of this fix described in #4493 (comment) still holds (most probably there is something to be fixed in the LB configuration because if this takes a long time to detect problems we are going to incur in errors on delete or on subsequent API calls)

k8s-triage-robot · 2021-10-19T16:42:35Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

randomvariable · 2021-11-02T14:19:30Z

/area control-plane

fabriziopandini · 2021-11-02T14:28:56Z

Given that this does not seem a problem and probably the issue in this thread is related to LB misconfigurations
We can eventually re-open if there are new evidences
/close

k8s-ci-robot · 2021-11-02T14:29:10Z

@fabriziopandini: Closing this issue.

In response to this:

Given that this does not seem a problem and probably the issue in this thread is related to LB misconfigurations
We can eventually re-open if there are new evidences
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. labels Apr 20, 2021

chymy closed this as completed Apr 21, 2021

k8s-ci-robot reopened this Apr 23, 2021

k8s-ci-robot changed the title ~~workcluster(one master + one node) upgrade~~ workcluster(one control plane + one node) upgrade Apr 26, 2021

vincepri removed the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Jul 6, 2021

k8s-ci-robot assigned sbueringer Jul 6, 2021

k8s-ci-robot added this to the v0.4 milestone Jul 6, 2021

k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jul 6, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 19, 2021

vincepri modified the milestones: v0.4, v1.1 Oct 22, 2021

k8s-ci-robot added the area/control-plane Issues or PRs related to control-plane lifecycle management label Nov 2, 2021

k8s-ci-robot closed this as completed Nov 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

workcluster(one control plane + one node) upgrade #4493

workcluster(one control plane + one node) upgrade #4493

chymy commented Apr 20, 2021

k8s-ci-robot commented Apr 20, 2021

sbueringer commented Apr 20, 2021 •

edited

Loading

chymy commented Apr 21, 2021

chymy commented Apr 23, 2021 •

edited

Loading

chymy commented Apr 23, 2021

sbueringer commented Apr 23, 2021

k8s-ci-robot commented Apr 23, 2021

chymy commented Apr 25, 2021

vincepri commented Apr 26, 2021

chymy commented May 26, 2021

vincepri commented Jul 6, 2021

sbueringer commented Jul 9, 2021

fabriziopandini commented Jul 9, 2021

chymy commented Jul 13, 2021

sbueringer commented Jul 13, 2021

chymy commented Jul 13, 2021

sbueringer commented Jul 13, 2021 •

edited

Loading

chymy commented Jul 13, 2021

sbueringer commented Jul 13, 2021

chymy commented Jul 13, 2021

fabriziopandini commented Jul 13, 2021 •

edited

Loading

sbueringer commented Jul 13, 2021 •

edited

Loading

sbueringer commented Jul 19, 2021

fabriziopandini commented Jul 21, 2021

k8s-triage-robot commented Oct 19, 2021

randomvariable commented Nov 2, 2021

fabriziopandini commented Nov 2, 2021

k8s-ci-robot commented Nov 2, 2021

workcluster(one control plane + one node) upgrade #4493

workcluster(one control plane + one node) upgrade #4493

Comments

chymy commented Apr 20, 2021

k8s-ci-robot commented Apr 20, 2021

sbueringer commented Apr 20, 2021 • edited Loading

chymy commented Apr 21, 2021

chymy commented Apr 23, 2021 • edited Loading

chymy commented Apr 23, 2021

sbueringer commented Apr 23, 2021

k8s-ci-robot commented Apr 23, 2021

chymy commented Apr 25, 2021

vincepri commented Apr 26, 2021

chymy commented May 26, 2021

vincepri commented Jul 6, 2021

sbueringer commented Jul 9, 2021

fabriziopandini commented Jul 9, 2021

chymy commented Jul 13, 2021

sbueringer commented Jul 13, 2021

chymy commented Jul 13, 2021

sbueringer commented Jul 13, 2021 • edited Loading

chymy commented Jul 13, 2021

sbueringer commented Jul 13, 2021

chymy commented Jul 13, 2021

fabriziopandini commented Jul 13, 2021 • edited Loading

sbueringer commented Jul 13, 2021 • edited Loading

sbueringer commented Jul 19, 2021

fabriziopandini commented Jul 21, 2021

k8s-triage-robot commented Oct 19, 2021

randomvariable commented Nov 2, 2021

fabriziopandini commented Nov 2, 2021

k8s-ci-robot commented Nov 2, 2021

sbueringer commented Apr 20, 2021 •

edited

Loading

chymy commented Apr 23, 2021 •

edited

Loading

sbueringer commented Jul 13, 2021 •

edited

Loading

fabriziopandini commented Jul 13, 2021 •

edited

Loading

sbueringer commented Jul 13, 2021 •

edited

Loading