Declarative Node Maintenance

[atiratree]

Enhancement Description

• One-line enhancement description (can be used as a release note): introduce a NodeMaintenance API to solve various node drain issues
• Kubernetes Enhancement Proposal: KEP-4212: Declarative Node Maintenance #4213
• Discussion Link: https://kubernetes.slack.com/archives/C18NZM5K9/p1690232891010319
• Primary contact (assignee): @atiratree
• Responsible SIGs: sig-apps
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.32
  • Beta release target (x.y):
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s): KEP-4212: Declarative Node Maintenance #4213
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[atiratree]

/sig apps

[npolshakova]

Hello @atiratree, 1.29 Enhancements team here! Is this enhancement targeting 1.29? If it is, can you follow the instructions here to opt in the enhancements and make sure the lead-opted-in label is set so it can get added to the tracking board? Thanks!

[sftim]

Title suggestion: “Declarative node maintenance”

(all KEPs are implicitly improvements, or at least they aim to be)

[atiratree]

Thanks for the suggestion. I have updated the title and the KEP.

[atiratree]

@npolshakova this still needs to be discussed with all the interested sigs before we can target this.

[atiratree]

this has been discussed and the KEP still needs some work and additional discussions, so we will try to target this for the next release

[alculquicondor]

/sig node

[sftim]

@atiratree, how can people best help move this work forward?

[atiratree]

I will revisit this one soon. I need to process the reviews and my personal notes.

[atiratree]

The KEP should be up to date now.

[sftim]

Some relevance to: kubernetes/website#44998 (about the existing docs for node maintenance)

[beekhof]

In case it helps, Red Hat has been shipping https://github.com/medik8s/node-maintenance-operator for ~5 years.
It almost certainly contains some OpenShift-isms, but I'm sure they can be addressed if there is interest.

[atiratree]

@beekhof I am aware of this operator and it was considered when designing this feature. Please also see https://github.com/kubernetes/enhancements/blob/cd1ea31e1c09c2f4e9f6a7f35821ff14f41a2f78/keps/sig-apps/4212-declarative-node-maintenance/README.md#out-of-tree-implementation

[atiratree]

The Declarative Node Maintenance KEP is becoming too complex and it is hard to capture all aspects and review everything in a single place. I have opened a second KEP just for the Evacuation API: #4563 and it will be a prerequisite for the Node Maintenance.

[sftim]

@beekhof if we did prototype node maintenance out of tree, with an alpha API group, we'd want to teach Kubernetes tooling about using it (for example, kubectl drain - but also many other in-project pieces).

To me, the value comes from that integration work more than from writing the actual controller. If you agree, we could look at rallying effort around an out-of-tree prototype. The integrations would remain valuable even after a move to in-tree implementation.

[soltysh]

/assign @atiratree

/label lead-opted-in
/stage alpha
/milestone v1.31

[atiratree]

@soltysh I would like to consider this for alpha in 1.32, not in this release. To get more feedback on the feature and to lead the way with the Evacuation API (dependency) first.

[soltysh]

/milestone clear

[dipesh-rawat]

Hello @soltysh @atiratree 👋, 1.31 Enhancements team here.

Now that this KEP is not targeting for release 1.31 (reference #4212 (comment)), should we also consider removing the lead-opted-in label too?

[soltysh]

Yeah, I guess you're right.

/remove-label lead-opted-in

[sftim]

Is there a baseline of detail that we can define and merge as provisional?

[soltysh]

This and #4563 where discussed heavily again during Monday's SIG-Apps call, with the general hesitation from SIG leads we're going to hold on with these efforts in favor of looking into extending PDB and eviction API.

[sreeram-venkitesh]

/label tracked/no

[sftim]

I'm tempted to propose an out-of-tree implementation (CRD, controller) and maybe kubectl plugin.
kubectl experimental-drain perhaps?

[kannon92]

Reading this, we are not considering this for 1.32?

[atiratree]

Not considering for 1.32. We have to get a broader consensus before we can accept this into the core. Please see https://groups.google.com/g/kubernetes-sig-architecture/c/Tb_3oDMAHrg for more details. I would like to build the consensus with the community, but did not have a time to pick up the topic yet. Hopefully we can do that soon.

We are also having discussions about the #4565 and will follow up on that in the next sig-apps meeting, which would help the NodeMaintenance feature.

[atiratree]

I'm tempted to propose an out-of-tree implementation (CRD, controller) and maybe kubectl plugin.
kubectl experimental-drain perhaps?

What would be the benefit when compared to the other drain solutions? Or extending the other drain solutions? Nevertheless, I think it is better to start with the discussions first.

[sftim]

What would be the benefit when compared to the other drain solutions? Or extending the other drain solutions?

Personally, I want to be able to initiate a declarative drain via kubectl. The only in-tree element there would be a change to kubectl drain.

[atiratree]

We cannot make it declarative unless we have the API and I am not positive we can accept 3rd party APIs into kubectl. But we could consider starting with the NodeMaintenance logic + Evacuation API in kubectl.

[sftim]

I am not positive we can accept 3rd party APIs into kubectl

Did you mean “out of tree”? The out-of-tree implementation I had in mine would still be part of Kubernetes, but not part of https://github.com/kubernetes/kubernetes (other than the kubectl support, which needs to be in-tree if not done using a plugin).

[atiratree]

Ah, yeah, that could work if sig-cli would accept it. However, if there is an API, there is less need for kubectl integration and just creating the object could be enough.