Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📖 Update book section regarding external etcd #2962

Merged
merged 2 commits into from
Apr 30, 2020
+54 −27
Merged

📖 Update book section regarding external etcd #2962

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6c31c0e
Update external etcd documentation
scottslowe Apr 24, 2020
1876bc0
Update etcd certificate discussion
scottslowe Apr 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 54 additions & 27 deletions bootstrap/kubeadm/docs/external-etcd.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,49 +1,76 @@
# Support for external etcd

Cluster API Bootstrap Provider Kubeadm supports using an external etcd cluster for your workload Kubernetes clusters.
Cluster API Bootstrap Provider Kubeadm supports using an external etcd cluster for your workload Kubernetes clusters.

## ⚠️ Warnings ⚠️

Before getting started you should be aware of the expectations that come with using an external etcd cluster.

* Cluster API is unable to manage any aspect of the external etcd cluster.
* Depending on how you configure your etcd nodes you may incur additional cloud costs in data transfer.
* As an example, cross availability zone traffic can cost money on cloud providers. You don't have to deploy etcd
across availability zones, but if you do please be aware of the costs.
* As an example, cross availability zone traffic can cost money on cloud providers. You don't have to deploy etcd across availability zones, but if you do please be aware of the costs.

## Getting started

To use this, you will need to create an etcd cluster and generate an apiserver-etcd-client key/pair.
This behaviour can be tested using [`kubeadm`](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/) and [`etcdadm`](https://github.com/kubernetes-sigs/etcdadm).
To use this, you will need to create an etcd cluster and generate an apiserver-etcd-client certificate and private key. This behaviour can be tested using [`kubeadm`](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/) and [`etcdadm`](https://github.com/kubernetes-sigs/etcdadm).

### Setting up etcd with kubeadm

CA certificates are required to setup etcd cluster.
If you already have a CA then the CA's `key` and `crt` must be copied to `/etc/kubernetes/pki/etcd/ca.crt` and `/etc/kubernetes/pki/etcd/ca.key`.
CA certificates are required to setup etcd cluster. If you already have a CA then the CA's `crt` and `key` must be copied to `/etc/kubernetes/pki/etcd/ca.crt` and `/etc/kubernetes/pki/etcd/ca.key`.

If you do not already have a CA then run command `kubeadm init phase certs etcd-ca`. This creates two files
If you do not already have a CA then run command `kubeadm init phase certs etcd-ca`. This creates two files:

* `/etc/kubernetes/pki/etcd/ca.crt`
* `/etc/kubernetes/pki/etcd/ca.key`
* `/etc/kubernetes/pki/etcd/ca.crt`
* `/etc/kubernetes/pki/etcd/ca.key`

These key/pair are used to sign etcd server, peer certificates and eventually apiserver-etcd client. More information on how to setup external etcd with kubeadm can be found [`here`](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/#setting-up-the-cluster).
This certificate and private key are used to sign etcd server, peer certificates and eventually apiserver-etcd-client certificate. More information on how to setup external etcd with kubeadm can be found [here](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/#setting-up-the-cluster).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kubeadm also uses the etcd CA to sign healthcheck-client.{crt|key}, which are more appropriate to have been called client... as the resulting TLS configuration is used for more actions than just "healthcheck". the healthcheck- part is residue from the days when kubeadm used a bash/etcdctl liveness probe for etcd server instances.

leaving it to you if you want to document it as e.g. just client certificates, or omit this part.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated it in 1876bc0, PTAL.


You would require files `/etc/kubernetes/pki/apiserver-etcd-client.key`, `/etc/kubernetes/pki/apiserver-etcd-client.crt` and `/etc/kubernetes/pki/etcd/server.crt` to setup etcd cluster. These are put in 2 secrets.
Once the etcd cluster is setup, you will need the following files from the etcd cluster:

```
# Kubernetes APIServer etcd client certificate
1. `/etc/kubernetes/pki/apiserver-etcd-client.crt` and `/etc/kubernetes/pki/apiserver-etcd-client.key`
2. `/etc/kubernetes/pki/etcd/ca.crt`

You'll use these files to create the necessary Secrets on the management cluster (see the "Creating the required Secrets" section).

### Setting up etcd with etcdadm (Alpha)
Copy link
Member

@neolit123 neolit123 Apr 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @dlipovetsky
i'm having flashbacks about a similar PR some time ago.
did we end up not wanting to have etcdadm docs in CAP yet;; i cannot recall?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

never mind, i see this was present already bellow.
@dlipovetsky PTAL if you can at the update.


`etcdadm` creates the CA if one does not exist, uses it to sign its server and peer certificates, and finally to sign the API server etcd client certificate. The CA's `crt` and `key` generated using `etcdadm` are stored in `/etc/etcd/pki/ca.crt` and `/etc/etcd/pki/ca.key`. `etcdadm` also generates a certificate for the API server etcd client; the certificate and private key are found at `/etc/etcd/pki/apiserver-etcd-client.crt` and `/etc/etcd/pki/apiserver-etcd-client.key`, respectively.

Once the etcd cluster has been bootstrapped using `etcdadm`, you will need the following files from the etcd cluster:

1. `/etc/etcd/pki/apiserver-etcd-client.crt` and `/etc/etcd/pki/apiserver-etcd-client.key`
2. `/etc/etcd/pki/etcd/ca.crt`

You'll use these files in the next section to create the necessary Secrets on the management cluster.

## Creating the required Secrets

Regardless of the method used to bootstrap the etcd cluster, you will need to use the certificates copied from the etcd cluster to create some [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-using-kubectl-create-secret) on the management cluster.

In the commands below to create the Secrets, substitute `$CLUSTER_NAME` with the name of the workload cluster to be created by CAPI, and substitute `$CLUSTER_NAMESPACE` with the name of the namespace where the workload cluster will be created. The namespace can be omitted if the workload cluster will be created in the default namespace.

First, you will need to create a Secret containing the API server etcd client certificate and key. This command assumes the certificate and private key are in the current directory; adjust your command accordingly if they are not:

```bash
# Kubernetes API server etcd client certificate and key
$ kubectl create secret tls $CLUSTER_NAME-apiserver-etcd-client \
--cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key /etc/kubernetes/pki/apiserver-etcd-client.crt \
--cert apiserver-etcd-client.crt \
--key apiserver-etcd-client.key \
--namespace $CLUSTER_NAMESPACE
```

Next, create a Secret for the etcd cluster's CA certificate. The `kubectl create secret tls` command requires _both_ a certificate and a key, but the key isn't needed by CAPI. Instead, use the `kubectl create secret generic` command, and note that the file containing the CA certificate **must** be named `tls.crt`:

```bash
# Etcd's CA crt file to validate the generated client certificates
$ kubectl create secret tls $CLUSTER_NAME-etcd --cert /etc/kubernetes/pki/etcd/server.crt \
$ kubectl create secret generic $CLUSTER_NAME-etcd \
--from-file tls.crt \
--namespace $CLUSTER_NAMESPACE
```
**Note:** Above command has key/pair base64 encoded by default.

**Note:** Alternatively you can base64 encode the files and put them in two secrets. The secrets
must be formatted as follows and the cert material must be base64 encoded:
**Note:** The above commands will base64 encode the certificate/key files by default.

Alternatively you can base64 encode the files and put them in two secrets. The secrets must be formatted as follows and the cert material must be base64 encoded:

```yaml
# Kubernetes APIServer etcd client certificate
Expand Down Expand Up @@ -77,15 +104,11 @@ data:
...
```

### Setting up etcd with etcdadm (Alpha)
`etcdadm` creates the CA if one does not exist, uses it to sign it's server and peer certificates, and finally to sign the apiserver etcd client certificate.
CA's `key` and `crt` generated using `etcdadm` are stored in `/etc/etcd/pki/apiserver-etcd-client.crt`, `/etc/etcd/pki/apiserver-etcd-client.key` and `/etc/etcd/pki/server.crt` .


Just like kubeadm, it is required to create 2 [`secrets`](https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-using-kubectl-create-secret) using these server and etcd client key/pair.
The Secrets must be created _before_ creating the workload cluster.

## Configuring CABPK
After that the rest is standard Kubeadm. Config your ClusterConfiguration as follows:

Once the Secrets are in place on the management cluster, the rest of the process leverages standard `kubeadm` configuration. Configure your ClusterConfiguration for the workload cluster as follows:

```yaml
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
Expand All @@ -106,4 +129,8 @@ spec:
... # other clusterConfiguration goes here
```

Create your cluster as normal!
Create your workload cluster as normal. The new workload cluster should use the configured external etcd nodes instead of creating co-located etcd Pods on the control plane nodes.

## Additional Notes/Caveats

* Depending on the provider, additional changes to the workload cluster's manifest may be necessary to ensure the new CAPI-managed nodes have connectivity to the existing etcd nodes. For example, on AWS you will need to leverage the `additionalSecurityGroups` field on the AWSMachine and/or AWSMachineTemplate objects to add the CAPI-managed nodes to a security group that has connectivity to the existing etcd cluster. Other mechanisms exist for other providers.
Copy link
Member

@neolit123 neolit123 Apr 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if someone likes to put their etcd cluster behind a LB this has the benefit of kubeadm not knowing the whole list of external member SANs. in theory this would allow the etcd cluster to grow / change without kubeadm knowing. unclear to me weather --etcd-servers for kube-apiservers can including only that LB DNS though. EDIT: actually --etcd-servers needs ip:port.

we don't have this in the kubeadm docs, so probably we need to add it there first.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It sounds like maybe we can omit a discussion of using an LB for etcd here until it has been added to other documentation. Is there an issue somewhere for addressing this in the kubeadm docs? Maybe I can help in some way.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It sounds like maybe we can omit a discussion of using an LB for etcd here

yes

Is there an issue somewhere for addressing this in the kubeadm docs

no, i don't think anyone has tested this on our side.

Maybe I can help in some way.

it's very low priority unless there is more demand. before documenting it, it needs an e2e test too using kinder.