From a00748ffe92e3babd9d7ece9ec35c0baef9b36ca Mon Sep 17 00:00:00 2001 From: Michael Gugino Date: Tue, 2 Jun 2020 16:39:08 -0400 Subject: [PATCH] CAEP: machine deletion phase hooks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Defines a set of annotations that can be applied to a machine which affect the linear progress of a machine’s lifecycle after a machine has been marked for deletion. These annotations are optional and may be applied during machine creation, sometime after machine creation by a user, or sometime after machine creation by another controller or application. --- .../20200602-machine-deletion-phase-hooks.md | 315 ++++++++++++++++++ 1 file changed, 315 insertions(+) create mode 100644 docs/proposals/20200602-machine-deletion-phase-hooks.md diff --git a/docs/proposals/20200602-machine-deletion-phase-hooks.md b/docs/proposals/20200602-machine-deletion-phase-hooks.md new file mode 100644 index 000000000000..6f2b41d20d76 --- /dev/null +++ b/docs/proposals/20200602-machine-deletion-phase-hooks.md @@ -0,0 +1,315 @@ +--- +title: Machine Deletion Phase Hooks +authors: + - "@michaelgugino" +reviewers: + - "@enxebre" + - "@vincepri" + - "@detiber" + - "@ncdc" +creation-date: 2020-06-02 +last-updated: 2020-06-02 +status: implementable +--- + +# Machine Deletion Phase Hooks + +## Table of Contents + +- [Machine Deletion Phase Hooks](#machine-deletion-phase-hooks) + - [Table of Contents](#table-of-contents) + - [Glossary](#glossary) + - [lifecycle hook](#lifecycle-hook) + - [deletion phase](#deletion-phase) + - [Summary](#summary) + - [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals/Future Work](#non-goalsfuture-work) + - [Proposal](#proposal) + - [User Stories](#user-stories) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Story 3](#story-3) + - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints) + - [Lifecycle Points](#lifecycle-points) + - [pre-drain](#pre-drain) + - [pre-terminate](#pre-terminate) + - [Annotation Form](#annotation-form) + - [lifecycle-point](#lifecycle-point) + - [hook-name](#hook-name) + - [owner (Optional)](#owner-optional) + - [Annotation Examples](#annotation-examples) + - [Changes to machine-controller](#changes-to-machine-controller) + - [Reconciliation](#reconciliation) + - [Hook failure](#hook-failure) + - [Hook ordering](#hook-ordering) + - [Owning Controller Design](#owning-controller-design) + - [Owning Controllers must](#owning-controllers-must) + - [Owning Controllers may](#owning-controllers-may) + - [Determining when to take action](#determining-when-to-take-action) + - [Failure Mode](#failure-mode) + - [Risks and Mitigations](#risks-and-mitigations) + - [Alternatives](#alternatives) + - [Custom Machine Controller](#custom-machine-controller) + - [Finalizers](#finalizers) + - [Status Field](#status-field) + - [Spec Field](#spec-field) + - [CRDs](#crds) + - [Upgrade Strategy](#upgrade-strategy) + - [Additional Details](#additional-details) + + +## Glossary + +Refer to the [Cluster API Book Glossary](https://cluster-api.sigs.k8s.io/reference/glossary.html). + +### lifecycle hook +A specific point in a machine's reconciliation lifecycle where execution of +normal machine-controller behavior is paused or modified. + +### deletion phase +Describes when a machine has been marked for deletion but is still present +in the API. Various actions happen during this phase, such as draining a node, +deleting an instance from a cloud provider, and deleting a node object. + +### Hook Implementing Controller (HIC) +The Hook Implementing Controller describes a controller, other than the +machine-controller, that adds, removes, and/or responds to a particular +lifecycle hook. Each lifecycle hook should have a single HIC, but an HIC +can optionally manage one or more hooks. + +## Summary + +Defines a set of annotations that can be applied to a machine which affect the +linear progress of a machine’s lifecycle after a machine has been marked for +deletion. These annotations are optional and may be applied during machine +creation, sometime after machine creation by a user, or sometime after machine +creation by another controller or application. + +## Motivation + +Allow custom and 3rd party components to easily interact with a machine or +related resources while that machine's reconciliation is temporarily paused. +This pause in reconciliation will allow these custom components to take action +after a machine has been marked for deletion, but prior to the machine being +drained and/or associated instance terminated. + +### Goals + +- Define an initial set of hook points for the deletion phase. +- Define an initial set and form of related annotations. +- Define basic expectations for a controller or process that responds to a +lifecycle hook. + + +### Non-Goals/Future Work + +- Create an exhaustive list of hooks; we can add more over time. +- Create new machine phases. +- Create a mechanism to signal what lifecycle point a machine is at currently. +- Dictate implementation of controllers that respond to the hooks. +- Implement ordering in the machine-controller. +- Require anyone use these hooks for normal machine operations, these are +strictly optional and for custom integrations only. + + +## Proposal + +- Utilize annotations to implement lifecycle hooks. +- Each lifecycle point can have 0 or more hooks. +- Hooks do not enforce ordering. +- Hooks found during machine reconciliation effectively pause reconciliation +until all hooks for that lifecycle point are removed from a machine's annotations. + + +### User Stories +#### Story 1 +(pre-terminate) As an operator, I would like to have the ability to perform +different actions between the time a machine is marked deleted in the api and +the time the machine is deleted from the cloud. + +For example, when replacing a control plane machine, ensure a new control +plane machine has been successfully created and joined to the cluster before +removing the instance of the deleted machine. This might be useful in case +there are disruptions during replacement and we need the disk of the existing +instance to perform some disaster recovery operation. This will also prevent +prolonged periods of having one fewer control plane host in the event the +replacement instance does not come up in a timely manner. + +#### Story 2 +(pre-drain) As an operator, I want the ability to utilize my own draining +controller instead of the logic built into the machine-controller. This will +allow me better flexibility and control over the lifecycle of workloads on each +node. + +#### Story 3 +(pre-drain) As an operator, when I am deleting a control plane machine for +maintenance reasons, I want to ensure my existing control plane machine is not +drained until after my replacement comes online. This will prevent protracted +periods of a missing control plane member if the replacement machine cannot be +created in a timely manner. + + +### Implementation Details/Notes/Constraints + +For each defined lifecycle point, one or more hooks may be applied as an annotation to the machine object. These annotations will pause reconciliation of a machine object until all hooks are resolved for that lifecycle point. The hooks should be managed by an Hook Implementing Controler or other external application, or +manually created and removed by an administrator. + +#### Lifecycle Points +##### pre-drain +`pre-drain.delete.hook.machine.cluster.x-k8s.io` + +Hooks defined at this point will prevent the machine-controller from draining a node after the machine-object has been marked for deletion until the hooks are removed. +##### pre-terminate +`pre-terminate.delete.hook.machine.cluster.x-k8s.io` + +Hooks defined at this point will prevent the machine-controller from +removing/terminating the instance in the cloud provider until the hooks are +removed. + +"pre-terminate" has been chosen over "pre-delete" because "terminate" is more +easily associated with an instance being removed from the cloud or +infrastructure, whereas "delete" is ambiguous as to the actual state of the +machine in its lifecycle. + + +#### Annotation Form +``` +.delete.hook.machine.cluster-api.x-k8s.io/: +``` + +##### lifecycle-point +This is the point in the lifecycle of reconciling a machine the annotation will have effect and pause the machine-controller. + +##### hook-name +Each hook should have a unique and descriptive name that describes in 1-3 words what the intent/reason for the hook is. Each hook name should be unique and managed by a single entity. + +##### owner (Optional) +Some information about who created or is otherwise in charge of managing the annotation. This might be a controller or a username to indicate an administrator applied the hook directly. + +##### Annotation Examples + +These examples are all hypothetical to illustrate what form annotations should +take. The names of of each hook and the respective controllers are fictional. + +pre-drain.hook.machine.cluster-api.x-k8s.io/migrate-important-app: my-app-migration-controller + +pre-terminate.hook.machine.cluster-api.x-k8s.io/backup-files: my-backup-controller + +pre-terminate.hook.machine.cluster-api.x-k8s.io/wait-for-storage-detach: my-custom-storage-detach-controller + +#### Changes to machine-controller +The machine-controller should check for the existence of 1 or more hooks at +specific points (lifecycle-points) during reconciliation. If a hook matching +the lifecycle-point is discovered, the machine-controller should stop +reconciling the machine. + +An example of where the pre-drain lifecycle-point might be implemented: +https://github.com/kubernetes-sigs/cluster-api/blob/30c377c0964efc789ab2f3f7361eb323003a7759/controllers/machine_controller.go#L270 + +##### Reconciliation +When a Hook Implementing Controller updates the machine, reconciliation will be +triggered, and the machine will continue reconciling as normal, unless another +hook is still present; there is no need to 'fail' the reconciliation to +enforce requeuing. + +When all hooks for a given lifecycle-point are removed, reconciliation +will continue as normal. + +##### Hook failure +The machine-controller should not timeout or otherwise consider the lifecycle +hook as 'failed.' Only the Hook Implementing Controller may decide to remove a +particular lifecycle hook to allow the machine-controller to progress past the +corresponding lifecycle-point. + +##### Hook ordering +The machine-controller will not attempt to enforce any ordering of hooks. + +Hook Implementing Controllers may devise a dependency-based enforcement mechanism +via whatever means HICs determine. Examples could be +using CRDs external to the machine-api, gRPC communications, or +additional annotations on the machine or other objects. + +#### Hook Implementing Controller Design +Hook Implementing Controller is the component that manages a particular +lifecycle hook. + +##### Hook Implementing Controllers must +* Watch machine objects and determine when an appropriate action must be taken. +* After completing the desired hook action, remove the hook annotation. +##### Hook Implementing Controllers may +* Watch machine objects and add a hook annotation as desired by the cluster +administrator. +* After completing a hook, set an addition annotation that indicates this +controller has finished for dependency ordering, or perform some other action +to signal other Hook Implementing Controllers that they may proceed. + +#### Determining when to take action + +An Hook Implementing Controller should watch machines and determine when is the +best time to take action. + +For example, if an HIC manages a lifecycle hook at the pre-drain lifecycle-point, +then that controller should take action immediately after a machine has a +DeletionTimestamp or enters the "Deleting" phase. + +Fine-tuned coordination is not possible at this time; eg, it's not +possible to execute a pre-terminate hook only after a node has been drained. +This is reserved for future work. + +##### Failure Mode +It is entirely up to the Hook Implementing Controller to determine when it is +prudent to remove a particular lifecycle hook. Some controllers may want to +'give up' after a certain time period, and others may want to block indefinitely. +Cluster operators should consider the characteristics of each controller before +utilizing them in their clusters. + + +### Risks and Mitigations + +* Annotation keys must conform to length limits: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set +* Requires well-behaved controllers and admins to keep things running +smoothly. Would be easy to disrupt machines with poor configuration. +* Troubleshooting problems may increase in complexity, but this is +mitigated mostly by the fact that these hooks are opt-in. Operators +will or should know they are consuming these hooks, but a future proliferation +of the cluster-api could result in these components being bundled as a +complete solution that operators just consume. To this end, we should +update any troubleshooting guides to check these hook points where possible. + + +## Alternatives + +### Custom Machine Controller +Require advanced users to fork and customize. This can already be done if someone chooses, so not much of a solution. + +### Finalizers +We define additional finalizers, but this really only implies the deletion lifecycle point. A misbehaving controller that +accidentally removes finalizers could have undesireable +effects. + +### Status Field +Harder for users to modify or set hooks during machine creation. How would a user remove a hook if a controller that is supposed to remove it is misbehaving? We’d probably need an annotation like ‘skip-hook-xyz’ or similar and that seems redundant to just using annotations in the first place + +### Spec Field +We probably don’t want other controllers dynamically adding and removing spec fields on an object. It’s not very declarative to utilize spec fields in that way. + +### CRDs +Seems like we’d need to sync information to and from a CR. There are different approaches to CRDs (1-to-1 mapping machine to CR, match labels, present/absent vs status fields) that each have their own drawbacks and are more complex to define and configure. + + +## Upgrade Strategy + +Nothing defined here should directly impact upgrades other than defining hooks that impact creation/deletion of a machine, generally. + +## Additional Details + +Fine-tuned timing of hooks is not possible at this time. + +In the future, it is possible to implement this timing via additional +machine phases, or possible "sub-phases" or some other mechanism +that might be appropriate. As stated in the non-goals, that is +not in scope at this time, and could be future work. + + +[community meeting]: https://docs.google.com/document/d/1Ys-DOR5UsgbMEeciuG0HOgDQc8kZsaWIWJeKJ1-UfbY