Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove usage of the kubeadm ClusterStatus object in Cluster API. #4423

Closed
neolit123 opened this issue Apr 1, 2021 · 15 comments · Fixed by #4643
Closed

remove usage of the kubeadm ClusterStatus object in Cluster API. #4423

neolit123 opened this issue Apr 1, 2021 · 15 comments · Fixed by #4643
Assignees
Labels
kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. kind/feature Categorizes issue or PR as related to a new feature. kind/release-blocking Issues or PRs that need to be closed before the next CAPI release lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Milestone

Comments

@neolit123
Copy link
Member

neolit123 commented Apr 1, 2021

ClusterStatus is an API structure present in kubeadm v1beta and v1beta2 that was used to track what API endpoints are available in the cluster. in 1.18 we deprecated it and implemented another method of doing the same, by applying annotations to the kube-apiserver pods. this already the default method in kubeadm, but ClusterStatus is still maintained as the second deprecated method.

the object is stored in the kubeadm-config CM.

https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2506-Remove-ClusterStatus-from-kubeadm-config

we had a discussion about this in the kubeadm office hours and @fabriziopandini mentioned that there could be impact on cluster API:
https://youtu.be/SZuK87bCfqI?t=1192

i see there there is some usage here:
https://github.com/kubernetes-sigs/cluster-api/search?q=clusterstatus

kubeadm has a plan to remove this API in v1beta3, which means Cluster API would not be able to use v1beta3 if it also uses ClusterStatus.

this issue is to migrate Cluster API to the new method of annotations and backport the change if needed.

@neolit123
Copy link
Member Author

/assign @fabriziopandini
please delegate if appropriate.

/kind deprecation feature
/priority important-soon

@k8s-ci-robot k8s-ci-robot added kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Apr 1, 2021
@fabriziopandini
Copy link
Member

@neolit123 Thanks for opening this issue. It is on my agenda for the next week

@fabriziopandini
Copy link
Member

fabriziopandini commented Apr 6, 2021

The implementation of an alternative approach of ClusterStatus in the Kubeadm ConfigMap landed in Kubernetes v1.18.

Given that and the fact that Cluster API must support the latest three Kubernetes releases for workload clusters (as of today v1.21, v1.20, v1.19), there are no blockers for dropping ClusterStatus object in kubeadm v1beta3 and as a consequence in Cluster API v1alpha4 types.

This will allow some cleanups in KCP

  • Drop logic for updating the ClusterStatus object in the kubeadm ConfigMap while scaling down
  • Drop logic for updating the ClusterStatus object in the kubeadm ConfigMap while remediating
  • Drop "catch-all" logic for reconciling ClusterStatus object in the kubeadm ConfigMap (reconcileEtcdMembers)

The downside of this change is that Cluster API / KCP v1alpha4 won't be able to manage Kubernetes clusters < v1.17, but this should not be a problem given that this release is out of the support skew.

Another important thing to notice is that dropping ClusterStatus object in kubeadm v1beta3 API could affect Cluster API v1alpha3 branch as well, de facto preventing CAPI v1alpha3 to manage Kubernetes clusters >= v1.22, unless we decide to implement & backport a workaround to make this possible.

@vincepri @detiber @CecileRobertMichon opinions ^^ ?

@CecileRobertMichon
Copy link
Contributor

The downside of this change is that Cluster API / KCP v1alpha4 won't be able to manage Kubernetes clusters < v1.17, but this should not be a problem given that this release is out of the support skew.

👍

Another important thing to notice is that dropping ClusterStatus object in kubeadm v1beta3 API could affect Cluster API v1alpha3 branch as well, de facto preventing CAPI v1alpha3 to manage Kubernetes clusters >= v1.22, unless we decide to implement & backport a workaround to make this possible.

Would that be breaking back compat rules for beta, ie. support for the last 3 release and/or up to a year ? Not sure if supporting new k8s versions is part of the contract.

@vincepri
Copy link
Member

vincepri commented Apr 6, 2021

The downside of this change is that Cluster API / KCP v1alpha4 won't be able to manage Kubernetes clusters < v1.17, but this should not be a problem given that this release is out of the support skew.

We should definitely document this bit and potentially block and upgrade to v0.4 if a cluster is older than 1.18

Would that be breaking back compat rules for beta, ie. support for the last 3 release and/or up to a year ? Not sure if supporting new k8s versions is part of the contract.

Up until today the Kubernetes version support for workload clusters has been pretty wide, we've allowed the supported version range to be quite wide. As an end user, I'd probably want some guarantees on which k8s versions are expected to be supported for a given series, and what are possible exceptions that have to make users upgrade.

If we maintain a stable release, we should always strive to support the latest Kubernetes versions, as these come with important patches most of the times, and it's generally a good practice to upgrade especially from versions that fall out of support.

Thoughts?

@sbueringer
Copy link
Member

sbueringer commented Apr 7, 2021

Another important thing to notice is that dropping ClusterStatus object in kubeadm v1beta3 API could affect Cluster API v1alpha3 branch as well, de facto preventing CAPI v1alpha3 to manage Kubernetes clusters >= v1.22, unless we decide to implement & backport a workaround to make this possible.

Not sure if we want to go there, but if I'm not mistaken we don't necessarily need kubeadm v1.22 to create a v1.22 Kubernetes clusters (or at least something like this worked in the past). So if other workarounds for v1alpha3 are more complicated, maybe we can use a kubeadm version < v1.22 to create Kubernetes clusters >= 1.22.

But not sure if that's a good idea.

EDIT: Forgot that the kubeadm/kubelet/kubectl versions are kept in sync and are coupled to the OS images, so this would be a pretty bad hack.

@fabriziopandini
Copy link
Member

As per discussion at the Apr 7th CAPI office hours:

  • OK for Cluster API / KCP v1alpha4 won't be able to manage Kubernetes clusters < v1.17,
  • Given the time rich between v1alpha4 release and Kubernetes v1.22 release, we should explore a possible workaround for allowing CAPI v1alpha3 to manage Kubernetes clusters >= v1.22

@fabriziopandini
Copy link
Member

@CecileRobertMichon @vincepri @detiber I have investigated a possible workaround for allowing CAPI v1alpha3 to manage Kubernetes clusters >= v1.22.

This is possible, but it requires dedicated PR targeting release-0.3 branch given the number of changes landed in this area for v1alpha4.

As a result starting from CAPI v0.3.16, it will be possible to install v1.22 clusters (older version can't do this given the changes in kubeadm)
If this is ok, I will start working on those changes

@vincepri
Copy link
Member

I'm fine with the approach described, @CecileRobertMichon @detiber thoughts?

@detiber
Copy link
Member

detiber commented Apr 13, 2021

@fabriziopandini awesome news, I'm a +1 for adding support to v0.3.x for k8s v1.22+

@CecileRobertMichon
Copy link
Contributor

SGTM, looks like this will be true for v0.3.17+ though as we'll most likely release v0.3.16 earlier.

@fabriziopandini
Copy link
Member

Ok, I will start working on this and try to get this into the v0.3.x train ASAP

@fabriziopandini
Copy link
Member

/assign
/lifecycle active

@CecileRobertMichon
Copy link
Contributor

/kind release-blocking

@k8s-ci-robot k8s-ci-robot added the kind/release-blocking Issues or PRs that need to be closed before the next CAPI release label May 24, 2021
@vincepri
Copy link
Member

/milestone v0.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. kind/feature Categorizes issue or PR as related to a new feature. kind/release-blocking Issues or PRs that need to be closed before the next CAPI release lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
7 participants