[WIP] Add document explaining certificates and PKI for Kubernetes #8232

liztio · 2018-04-30T20:40:16Z

This file is intended to help people understand the PKI around Kubernetes and how it can be configured when not delegating management to automated scripts.

I don't need to be on the approvers list, I just didn't know who else to put there.

liztio · 2018-04-30T20:41:17Z

/cc @Bradamant3

k8sio-netlify-preview-bot · 2018-04-30T20:42:26Z

Deploy preview for kubernetes-io-master-staging failed.

Built with commit b1f3cd3

https://app.netlify.com/sites/kubernetes-io-master-staging/deploys/5b6d97b6fdd72a5ddc772fc8

Bradamant3 · 2018-04-30T20:44:00Z

/assign

liztio · 2018-04-30T20:44:55Z

Needs inroads from existing documentation as well.

Bradamant3 · 2018-04-30T21:46:23Z

Travis CI build is failing bc TOC entry is missing -- this is by design at this point. I told Liz I'd work with her to figure out the best place to put it -- not immediately obvious. Incoming links per previous comment also important, yes. Modifying PR title and putting hold accordingly

/hold

Bradamant3 · 2018-04-30T21:47:02Z

/hold cancel

tengqm · 2018-05-01T07:22:44Z

The content of this page is mostly about kubeadm although the text sometimes is trying to make kubeadm just an example. I'd suggest this to be moved over to the kubeadm subtopic. The certificates topic is more than what kubeadm has implemented/supported, and there are many cases where people deploy their clusters using other tools. If we believe the generic concepts in the page can be extracted out as a top level reference topic, please do it.

liztio · 2018-05-01T13:59:41Z

@tengqm if you've got suggestions about how this could be made more generally applicable I'd love to hear them!

detiber

Great work on putting this together.

detiber · 2018-05-01T13:24:29Z

docs/reference/certificates.md

+
+# Introduction
+
+A tool like [kubeadm][kubeadm] can be used to generate your configuration for you.


Did you mean to use certificates here instead of configuration?

detiber · 2018-05-01T13:26:31Z

docs/reference/certificates.md

+1. Client certificates for the kubelet to authenticate to the master
+2. Server certificate for the api server endpoint
+3. Client certificates for administrators of the cluster authenticating to the API server
+4. Client certificates for service accounts authenticating to the API server.


This is being a bit pedantic, but technically the service accounts are using a generated public/private keypair only.

Certificates are also used for the following:

client certificate for the api server to talk to the kubelets

client certificate for the api server to talk to etcd

client certificate/kubeconfig for controller manager to talk to api server

client certificate/kubeconfig for scheduler to talk to api server

It may be good to mention the use of the front-proxy certs here as well, which are used to enable extension api servers as described here: https://kubernetes.io/docs/concepts/api-extension/apiserver-aggregation/

detiber · 2018-05-01T13:27:05Z

docs/reference/certificates.md

+
+## Where they're used
+
+Kubernetes uses kubeadm for four primary things:


s/kubeadm/certificates/

detiber · 2018-05-01T13:32:41Z

docs/reference/certificates.md

+## Where they're used
+
+Kubernetes uses kubeadm for four primary things:
+1. Client certificates for the kubelet to authenticate to the master


s/master/api server/

detiber · 2018-05-01T13:46:43Z

docs/reference/certificates.md

+3. Client certificates for administrators of the cluster authenticating to the API server
+4. Client certificates for service accounts authenticating to the API server.
+
+In addition, etcd uses mutual TLS for authenticating peers (in an HA cluster).$


s/peers/clients and peers/

detiber · 2018-05-01T13:50:13Z

docs/reference/certificates.md

+
+The CAs needed for this to work are:
+
+| path                   | CN                        | description                      |


should this be "path relative to /etc/kubernetes/pki" instead?

s/CN/default CN/

The actual CN used should not matter for the CAs

detiber · 2018-05-01T13:52:17Z

docs/reference/certificates.md

+| CN                            | Parent CA                 | O (in Subject) | kind   | hosts (SAN)                                 |
+|-------------------------------|---------------------------|----------------|--------|---------------------------------------------|
+| kube-etcd                     | etcd-ca                   |                | server | localhost, 127.0.0.1                        |
+| kube-etcd-peel                | etcd-ca                   |                | peer   | <hostname>, <Host IP>, localhost, 127.0.0.1 |


s/peel/peer/

detiber · 2018-05-01T13:59:33Z

docs/reference/certificates.md

+
+[usage]: https://tools.ietf.org/html/rfc5280#section-4.2.1.3
+[kubeadm]: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/ 
+[proxy]: https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#discovering-builtin-services


The front-proxy CA/certs are used for https://kubernetes.io/docs/concepts/api-extension/apiserver-aggregation/

tengqm · 2018-05-02T02:22:18Z

@liztio My suggestion is as follows:

Extract the generic contents about certificates into either one of the following pages:
- https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/
- https://kubernetes.io/docs/concepts/cluster-administration/certificates/
Move rest into a subtopic of https://kubernetes.io/docs/reference/setup-tools/kubeadm/ because they are specific to how kubeadm is manipulating the certificates.

zacharysarah · 2018-05-11T18:39:55Z

@liztio 👋 We've migrated the site's framework to Hugo. Please re-open this PR with changes to files in their post-migration location (/content/en/docs/...) and reference this PR in the new PR's description.

Bradamant3 · 2018-05-24T14:13:56Z

Reopening to remind me to help with this one as promised a while ago. PLEASE DO NOT CLOSE AGAIN.

Bradamant3 · 2018-05-25T12:09:20Z

possible places to link from:
https://kubernetes.io/docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm/

detiber

Overall this is looking great :)

detiber · 2018-06-14T16:37:06Z

docs/reference/certificates.md

+8. Client and server certificates for the [front-proxy][proxy]
+
+
+In addition, etcd uses mutual TLS for authenticating clients and peers (in an HA cluster).


Since we are generated certificates for the peer regardless of HA/non-HA, we can probably just remove the '(in a HA cluster)' qualifier here. Currently this reads a bit off to me because it's not clear to me that '(in a HA cluster)' only applies to the peer usage.

detiber · 2018-06-14T17:28:57Z

docs/reference/certificates.md

+
+| Default CN                    | Parent CA                 | O (in Subject) | kind   | hosts (SAN)                                 |
+|-------------------------------|---------------------------|----------------|--------|---------------------------------------------|
+| kube-etcd                     | etcd-ca                   |                | server | localhost, 127.0.0.1                        |


Since v1.11 now uses etcd 3.2.x, we currently have to create the etcd serving certificate with both server and client usages. Upstream etcd issue tracking this is here: etcd-io/etcd#9785

detiber · 2018-06-14T17:32:14Z

docs/reference/certificates.md

+|-------------------------------|---------------------------|----------------|--------|---------------------------------------------|
+| kube-etcd                     | etcd-ca                   |                | server | localhost, 127.0.0.1                        |
+| kube-etcd-peer                | etcd-ca                   |                | peer   | <hostname>, <Host IP>, localhost, 127.0.0.1 |
+| kube-etcd-healthcheck-client  | etcd-ca                   | system:masters | peer   |                                             |


kind for the kube-etcd-healthcheck-client cert should be client

Where will the kube-etcd-healthcheck-client CN be used? And it acts as a system:masters subject.

It is used by the healthcheck for the etcd static pod deployed by kubeadm.

This is currently how the kube-etcd-healthcheck-client cert is created using kubeadm today.

The system:masters subject here is a bit misleading and isn't necessary. Since this cert is created using the etcd CA it does not grant any additional permissions above and beyond any other cert that is created from that CA since we aren't enabling RBAC for etcd (at least yet).

detiber · 2018-06-14T17:37:30Z

docs/reference/certificates.md

+| kube-etcd-healthcheck-client  | etcd-ca                   | system:masters | peer   |                                             |
+| kube-apiserver-etcd-client    | etcd-ca                   | system:masters | client |                                             |
+| kube-apiserver                | kubernetes-ca             |                | server | <hostname>, <Host IP>, <advertise IP> , [1] |
+| kube-apiserver-client         | kubernetes-ca             | system:master  | client |                                             |


I think kube-apiserver-client might be a typo here. I'm not sure if it is extraneous or should be named something else, though.

detiber · 2018-06-14T17:40:53Z

docs/reference/certificates.md

+| filename                | credential name            | Default CN                     | O (in Subject) |
+|-------------------------|----------------------------|--------------------------------|----------------|
+| admin.conf              | default-admin              | kubernetes-admin               | system:masters |
+| kubelet.conf            | default-auth               | system:node:<hostname>         | system:nodes   |


The credential name for kubelet.conf should be system:node:<nodename>

detiber · 2018-06-14T17:43:35Z

docs/reference/certificates.md

+KUBECONFIG=<path> kubectl config set-cluster default-cluster --server=https://<host ip>:6443 --certificate-authority <path to kubernetes ca> --embed-certs
+KUBECONFIG=<path> kubectl config set-credentials <credential-name> --client-key <path-to-key>.pem --client-certificate <path-to-cert>.pem --embed-certs
+KUBECONFIG=<path> kubectl config set-context default-system --cluster default-cluster --user <credential-name>
+KUBECONFIG=<path> kubectl config use-context default-system


Path is used here, but appears to be referenced below.

detiber · 2018-06-14T17:44:57Z

docs/reference/certificates.md

+| default-admin              | admin.conf              | kubectl                 | used by a user to administer the cluster                    |
+| default-auth               | kubelet.conf            | kubelet                 | one will be needed for every node in the cluster.           |
+| default-controller-manager | controller-manager.conf | kube-controller-manager | add to manifest in `manifests/kube-controller-manager.yaml` |
+| default-scheduler          | scheduler.conf          | kube-scheduler          | add to manifest in `manifests/kube-scheduler.yaml`          |


I'd suggest adding path to the above table and removing name from this table.

chenopis · 2018-07-20T00:33:15Z

@liztio Please respond to the remaining feedback. Thx

k8s-ci-robot · 2018-08-02T20:12:55Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: bradamant3

If they are not already assigned, you can assign the PR to them by writing /assign @bradamant3 in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

liztio · 2018-08-02T20:13:05Z

@chenopis done, sorry it took so long!

Bradamant3 · 2018-08-10T13:52:03Z

added WIP to make sure we don't accidentally merge.

Pushed copyedit commit to this version so that @liztio can take a look. I'll then push up new PR based on hugo from their fork. N.B. this version does not resolve the links markup issue in Hugo. (not needed for content review)

Bradamant3 · 2018-08-10T13:53:07Z

docs/reference/certificates.md

-In that case, you will also need to make all of these:
+| ca.crt,key             | kubernetes-ca             | Kubernetes general CA            |
+| etcd/ca.crt,key        | etcd-ca                   | For all etcd-related functions   |
+| front-proxy-ca.crt,key | kubernetes-front-proxy-ca | For the [front-end proxy][proxy] |


It looked as though this and the previous line somehow got reversed in the version that I edited. If I've misunderstood something obscure, my apologies (but if I've misunderstood, we should explain a bit more too).

Bradamant3 · 2018-08-10T13:53:41Z

docs/reference/certificates.md

 | kube-apiserver-kubelet-client | kubernetes-ca             | system:masters | client                                 |                                             |
 | front-proxy-client            | kubernetes-front-proxy-ca |                | client                                 |                                             |

+[1]: `kubernetes`, `kubernetes.default`, `kubernetes.default.svc`, `kubernetes.default.svc.cluster`, `kubernetes.default.svc.cluster.local`


Changed markup for this and other notes because they weren't rendering at all in the original.

Bradamant3 · 2018-08-13T19:24:16Z

Finally closing unmerged bc we at long last have the Hugo PR.

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 30, 2018

k8s-ci-robot requested review from tengqm and zhangxiaoyu-zidif April 30, 2018 20:40

k8s-ci-robot requested a review from Bradamant3 April 30, 2018 20:41

liztio mentioned this pull request Apr 30, 2018

Example configs for using external Certificates kubernetes/kubeadm#638

Closed

k8s-ci-robot assigned Bradamant3 Apr 30, 2018

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 30, 2018

Bradamant3 changed the title ~~Add document explaining certificates and PKI for Kubernetes~~ [WIP] Add document explaining certificates and PKI for Kubernetes Apr 30, 2018

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 30, 2018

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 30, 2018

detiber reviewed May 1, 2018

View reviewed changes

Add document explaining certificates and PKI for Kubernetes

15f7665

liztio force-pushed the cert-docs branch from e2a9bd7 to 15f7665 Compare May 1, 2018 14:23

zacharysarah closed this May 11, 2018

Bradamant3 reopened this May 24, 2018

zacharysarah added the Docs Review: Open Issues label May 30, 2018

timothysc changed the title ~~[WIP] Add document explaining certificates and PKI for Kubernetes~~ Add document explaining certificates and PKI for Kubernetes Jun 13, 2018

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 13, 2018

detiber reviewed Jun 14, 2018

View reviewed changes

Docs review feedback

0be4a24

copyedits for clarity, template, partial hugo markdown

b1f3cd3

Bradamant3 changed the title ~~Add document explaining certificates and PKI for Kubernetes~~ [WIP] Add document explaining certificates and PKI for Kubernetes Aug 10, 2018

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 10, 2018

Bradamant3 reviewed Aug 10, 2018

View reviewed changes

Bradamant3 mentioned this pull request Aug 13, 2018

rebase certificates PR on Hugo master #9842

Merged

Bradamant3 closed this Aug 13, 2018


		# Introduction

		A tool like [kubeadm][kubeadm] can be used to generate your configuration for you.


		## Where they're used

		Kubernetes uses kubeadm for four primary things:


		The CAs needed for this to work are:

		\| path \| CN \| description \|

		8. Client and server certificates for the [front-proxy][proxy]


		In addition, etcd uses mutual TLS for authenticating clients and peers (in an HA cluster).

[WIP] Add document explaining certificates and PKI for Kubernetes #8232

[WIP] Add document explaining certificates and PKI for Kubernetes #8232

Conversation

liztio commented Apr 30, 2018 • edited by Bradamant3 Loading

liztio commented Apr 30, 2018

k8sio-netlify-preview-bot commented Apr 30, 2018 • edited Loading

Bradamant3 commented Apr 30, 2018

liztio commented Apr 30, 2018

Bradamant3 commented Apr 30, 2018

Bradamant3 commented Apr 30, 2018

tengqm commented May 1, 2018

liztio commented May 1, 2018

detiber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tengqm commented May 2, 2018

zacharysarah commented May 11, 2018

Bradamant3 commented May 24, 2018 • edited Loading

Bradamant3 commented May 25, 2018

detiber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JinsYin Jun 19, 2018 • edited Loading

Choose a reason for hiding this comment

detiber Jun 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenopis commented Jul 20, 2018

k8s-ci-robot commented Aug 2, 2018

liztio commented Aug 2, 2018

Bradamant3 commented Aug 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bradamant3 commented Aug 13, 2018

liztio commented Apr 30, 2018 •

edited by Bradamant3

Loading

k8sio-netlify-preview-bot commented Apr 30, 2018 •

edited

Loading

Bradamant3 commented May 24, 2018 •

edited

Loading

JinsYin Jun 19, 2018 •

edited

Loading

detiber Jun 19, 2018 •

edited

Loading