ETCD snapshot/restore support

[musaprg]

User Story

As an operator, I'd like to maintain the etcd snapshot/restore functionalities with Cluster API (KubeadmControlPlane).

Detailed Description

The etcd snapshot and restore are usually crucial for administrators. We can achieve those functionalities by using community-provided operators (e.g., etcd-operator, which is already archived though...) or etcdctl directly. However, sometimes restore tasks should be considered as one of the cluster lifecycles since it requires to stop/start kube-apiserver before/after restoring. It would be nice to provide the etcd snapshot/restore functionalities by the CAPI side so that we can easily maintain them.

(I couldn't find any discussions related to this except for #7399, so I filed this topic as a new issue. Please let me know if there are any places where we already have this kind of discussion.)

Anything else you would like to add:

(TBD)

Related Issues/PRs

• External etcd lifecycle support #7399
• 📖 CAEP to add support for managed external etcd clusters in CAPI #4659

/kind feature

[killianmuldoon]

Thanks @musaprg - this is a really interesting idea IMO. I think it go a long way toward helping out in disaster recovery scenarios.

A couple of questions:

1. Do you see this as only for workload clusters? Or for the management cluster?
2. Do you think standalone etcd is a necessary pre-requisite for this?
3. Would this require some kind of external storage in order to host the snapshot?

It would be great to get doc together with answers to some of these questions and, an overall problem statement and a start on implementation details to get people who would be interested in working on this feature involved.

[musaprg]

@killianmuldoon Thank you for asking. I'd like to prepare more detailed documents for this topic, but first, let me answer my opinions on the questions briefly.

1. Do you see this as only for workload clusters? Or for the management cluster?

I'm currently considering only workload clusters.

2. Do you think standalone etcd is a necessary pre-requisite for this?

I don't know if I could understand correctly the meaning of "standalone etcd", but I'm assuming etcd nodes that CAPI knows. IMO it doesn't matter how etcd nodes are running if they can be accessible from the management cluster.

3. Would this require some kind of external storage to host the snapshot?

It could have several options, but I don't think it's required. IMO possible destinations could be the following ones.

1. Object Storage (e.g., Amazon S3), which is external storage.
2. Persistent Volume on the management cluster, which is an in-cluster resource.

[musaprg]

I don't think it would be better to depend on something outside the Kubernetes ecosystem, so persistent volume would be the better candidate for the destination.

[fabriziopandini]

/triage accepted
I agree with @killianmuldoon this is interesting. I have got the impression this requires a small proposal defining scope and limitations, mostly because CAPI assumes bootstrap/control plane providers to be pluggable (and they are responsible both for etcd and control plane components)

[musaprg]

/assign

[enxebre]

As an operator, I'd like to maintain the etcd snapshot/restore functionalities with Cluster API (KubeadmControlPlane).

It'd be interesting to explore having these ability decoupled from KubeadmControlPlane or composable, so more control plane implementations could share it.

[Heiko-san]

Same problem here... It would be nice if cluster-api would provide a native feature for etcd snapshots (Which in my opinion is absolutely crucial! This is why I can't believe this was never thought of...)

We actually tried to implement it ourselves, but weren't very successful, since the etcd pod lacks really everything (We can do a etcdctl snapshot save in the pod, but can't obtain the file since kubectl cp complains about missing tar ...).

I couldn't find any helpful documentation on the topic, either. Is there something existing? There has to be a way to get snapshots from that pods, I guess...

[kfox1111]

Yeah, that broke a while back. I've been working around it by snapshotting to a host mount, then pulling it off the host. Less then ideal...

[musaprg]

We actually tried to implement it ourselves, but weren't very successful, since the etcd pod lacks really everything (We can do a etcdctl snapshot save in the pod, but can't obtain the file since kubectl cp complains about missing tar ...).

We currently create another pod that uses an image with required utilities (etcdctl, aws-cli, etc.) to create/upload snapshots into S3-compatible storages apart from etcd pods. It could be much easier than pulling it from the pod's local storage, but yes I'm thinking it would be nice if CAPI etcd snapshot support doesn't require any external storages.

[fabriziopandini]

Adding kind/proposal because I think we should figure out if and how to make this work with different bootstrap/control-plane providers (who are ultimately responsible for defining how etcd and api server are run) and/or if and how to make this work with different types of storage (which can or cannot be related to the infrastructure provider in use).
In other words, considering the CAPI provider model the issue is not to make this work, but to make it work in a generic way or to define a clear contract around the use cases it supports - and eventually how to overcome those limitations on follow up iterations -.

/kind proposal

[fabriziopandini]

/priority backlog

[fabriziopandini]

The Cluster API project currently lacks enough active contributors to adequately respond to all issues and PRs.

Also most probably this should fall under SIG etcd / the ongoing discussion about a community maintained etcd-operator

/close

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

The Cluster API project currently lacks enough active contributors to adequately respond to all issues and PRs.

Also most probably this should fall under SIG etcd / the ongoing discussion about a community maintained etcd-operator

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.