Feature/kubeadm 594 etcd TLS on init/upgrade #57415

stealthybox · 2017-12-19T18:06:14Z

What this PR does / why we need it:
On kubeadm init/kubeadm upgrade, this PR generates certificates for securing local etcd:

etcd serving cert
etcd peer cert
apiserver etcd client cert

Flags and hostMounts are added to the etcd and apiserver static-pods to load these certs.
For connections to etcd, https is now used in favor of http and tests have been added/updated.

Etcd only listens on localhost, so the serving cert SAN defaults to DNS:localhost,IP:127.0.0.1.
The etcd peer cert has SANs for <hostname>,<api-advertise-address>, but is unused.

New kubeadm config options, Etcd.ServerCertSANs and Etcd.PeerCertSANs, are used for user additions to the default certificate SANs for the etcd server and peer certs.

This feature continues to utilize the existence of MasterConfiguration.Etcd.Endpoints as a feature gate for external-etcd.
If the user passes flags to configure Etcd.{CAFile,CertFile,KeyFile} but they omit Endpoints, these flags will be unused, and a warning is printed.

New phase commands:

kubeadm alpha phase certs etcd-server
kubeadm alpha phase certs etcd-peer
kubeadm alpha phase certs apiserver-etcd-client

Which issue(s) this PR fixes
Fixes kubernetes/kubeadm#594

Special notes for your reviewer:

on the master

these should fail:

curl localhost:2379/v2/keys  # no output
curl --cacert /etc/kubernetes/pki/ca.crt https://localhost:2379/v2/keys  # handshake error

these should succeed:

cd /etc/kubernetes/pki
curl --cacert ca.crt --cert apiserver-etcd-client.crt --key apiserver-etcd-client.key https://localhost:2379/v2/keys

Release note:

On cluster provision or upgrade, kubeadm now generates certs and secures all connections to the etcd static-pod with mTLS.

stealthybox · 2017-12-19T18:11:09Z

/assign @luxas

luxas · 2017-12-22T13:56:35Z

/assign @kad @fabriziopandini
for initial review

0xmichalis · 2017-12-28T00:51:08Z

cmd/kubeadm/app/phases/certs/doc.go

@@ -36,6 +36,12 @@ package certs
 		 - apiserver.key
 		 - apiserver-kubelet-client.crt
 		 - apiserver-kubelet-client.key
+		 - etcd.crt


You also need to update INPUTS to include EtcdCertSANs.

Thanks. I've updated the doc-strings for both the SAN fields.

0xmichalis · 2017-12-28T01:00:14Z

cmd/kubeadm/app/phases/certs/certs.go

+			altNames.IPs = append(altNames.IPs, ip)
+		} else if len(validation.IsDNS1123Subdomain(altname)) == 0 {
+			altNames.DNSNames = append(altNames.DNSNames, altname)
+		}


ux enhancement: log when this is not a valid SAN

0xmichalis · 2017-12-28T01:01:48Z

cmd/kubeadm/app/phases/certs/certs.go

+			altNames.IPs = append(altNames.IPs, ip)
+		} else if len(validation.IsDNS1123Subdomain(altname)) == 0 {
+			altNames.DNSNames = append(altNames.DNSNames, altname)
+		}


same comment as above

stealthybox · 2017-12-29T08:12:38Z

@Kargakis, thanks for the suggestions -- changes are ready for another review.
The SAN warnings look like this:

[certificates] WARNING: 'hello-&&' was not added to the 'apiserver.crt' SAN, because it is not a valid IP or RFC-1123 compliant DNS entry
[certificates] WARNING: '129.2.3.L' was not added to the 'apiserver.crt' SAN, because it is not a valid IP or RFC-1123 compliant DNS entry

I implemented them for the API Server as well.

xiangpengzhao

some nits.

Another concern is that we may also want to backup etcd cert and key files as what is done for apiserver.
FYI:
#55998
kubernetes/kubeadm#548

kubernetes/cmd/kubeadm/app/phases/upgrade/postupgrade.go

Lines 86 to 99 in 6244ff3

    
           certAndKeyDir := kubeadmapiext.DefaultCertificatesDir 
        
           shouldBackup, err := shouldBackupAPIServerCertAndKey(certAndKeyDir, newK8sVer) 
        
           // Don't fail the upgrade phase if failing to determine to backup kube-apiserver cert and key. 
        
           if err != nil { 
        
           	fmt.Printf("[postupgrade] WARNING: failed to determine to backup kube-apiserver cert and key: %v", err) 
        
           } else if shouldBackup { 
        
           	// Don't fail the upgrade phase if failing to backup kube-apiserver cert and key. 
        
           	if err := backupAPIServerCertAndKey(certAndKeyDir); err != nil { 
        
           		fmt.Printf("[postupgrade] WARNING: failed to backup kube-apiserver cert and key: %v", err) 
        
           	} 
        
           	if err := certsphase.CreateAPIServerCertAndKeyFiles(cfg); err != nil { 
        
           		errs = append(errs, err) 
        
           	} 
        
           }

xiangpengzhao · 2018-01-09T01:47:01Z

cmd/kubeadm/app/phases/certs/certs.go

+// It assumes the cluster CA certificate and key file exist in the CertificatesDir
+func CreateEtcdCertAndKeyFiles(cfg *kubeadmapi.MasterConfiguration) error {
+
+	caCert, caKey, err := loadCertificateAuthorithy(cfg.CertificatesDir, kubeadmconstants.CACertAndKeyBaseName)


s/loadCertificateAuthorithy/loadCertificateAuthority ?

This requires a refactor -- I'll try it out as an additional commit.

Pushed up a change for this :)
Easy because it's a private function

xiangpengzhao · 2018-01-09T01:48:55Z

cmd/kubeadm/app/phases/certs/certs.go

@@ -230,6 +305,64 @@ func NewAPIServerKubeletClientCertAndKey(caCert *x509.Certificate, caKey *rsa.Pr
 	return apiClientCert, apiClientKey, nil
 }

+// NewEtcdCertAndKey generate CA certificate for etcd, signed by the given CA.


s/generate/generates

not going to do this in this PR -- all of the functions have similar phrasing for their docstrings

xiangpengzhao · 2018-01-09T01:50:22Z

cmd/kubeadm/app/phases/certs/certs.go

+	return etcdCert, etcdKey, nil
+}
+
+// NewEtcdPeerCertAndKey generate CA certificate for etcd peering, signed by the given CA.


s/generate/generates

not going to do this in this PR -- all of the functions have similar phrasing for their docstrings

xiangpengzhao · 2018-01-09T01:50:39Z

cmd/kubeadm/app/phases/certs/certs.go

+	return etcdPeerCert, etcdPeerKey, nil
+}
+
+// NewAPIServerEtcdClientCertAndKey generate CA certificate for the apiservers to connect to etcd securely, signed by the given CA.


s/generate/generates

not going to do this in this PR -- all of the functions have similar phrasing for their docstrings

timothysc · 2018-01-09T15:57:24Z

/cc @mattmoyer

xiangpengzhao · 2018-01-09T17:45:18Z

Also, as is mentioned in https://docs.google.com/document/d/19eei3oly2BqlgWFcqMtrp9_Ktlb3cNlSN8ZLntSt76I/edit#

During the upgrade case from 1.9 → 1.10 we’d need to generate the new certs, but that can easily be achieved

stealthybox · 2018-01-09T17:56:08Z

@xiangpengzhao With this PR, the certs are generated on upgrade when they don't exist. It doesn't discriminate between versions. Is this the behavior we want?

This cert renewal feature is something we didn't think about. We can implement that.
I suppose you backup the certs and re-use the CA to generate the new certs?
How long is the CA good for? I haven't read the code for this.

timothysc · 2018-01-09T21:54:56Z

I'll dig through pretty soon, but as a general rule, please squash commits.

xiangpengzhao · 2018-01-12T03:31:32Z

With this PR, the certs are generated on upgrade when they don't exist. It doesn't discriminate between versions. Is this the behavior we want?

@stealthybox yeah correct, this PR will cover this case "During the upgrade case from 1.9 → 1.10 we’d need to generate the new certs, but that can easily be achieved"

This cert renewal feature is something we didn't think about. We can implement that.

We can implement it in a follow-up PR.

I suppose you backup the certs and re-use the CA to generate the new certs?

correct.

How long is the CA good for? I haven't read the code for this.

IMO, one year would be ok (it's the default value?). And so all the certs can have the same validity duration.

jamiehannaford

made a first pass at review

jamiehannaford · 2018-01-12T17:34:34Z

cmd/kubeadm/app/apis/kubeadm/types.go

@@ -52,6 +52,10 @@ type MasterConfiguration struct {

 	// APIServerCertSANs sets extra Subject Alternative Names for the API Server signing cert
 	APIServerCertSANs []string
+
+	// EtcdCertSANs sets extra Subject Alternative Names for the etcd server and peer signing certs
+	EtcdCertSANs []string


will there ever be a case where SANs for peer and server will be different?

It's possible to have them listen on different interfaces.
I guessed it would be okay to just have a union of all of the needed SANs for peers and the server, but I could be wrong.

We could add another config for this.
I'm thinking it would be best to just add EtcdPeerCertSANs []string and not do anything fancy.
This makes the config more verbose, but it's flexible.

Do you think this is correct?

Renamed this to EtcdServerCertSANs and added EtcdPeerCertSANs

jamiehannaford · 2018-01-12T17:35:03Z

cmd/kubeadm/app/cmd/phases/certs.go

+	etcdCertLongDesc = fmt.Sprintf(normalizer.LongDesc(`
+		Generates the etcd serving certificate and key and saves them into %s and %s files.
+
+		The certificate includes default subject alternative names and additional sans eventually provided by the user;


fixed on all 3 (api, etcd, etcdPeer) -- thanks

jamiehannaford · 2018-01-12T17:36:10Z

cmd/kubeadm/app/cmd/phases/certs.go

@@ -157,6 +182,24 @@ func getCertsSubCommands(defaultKubernetesVersion string) []*cobra.Command {
 			long:    apiServerKubeletCertLongDesc,
 			cmdFunc: certsphase.CreateAPIServerKubeletClientCertAndKeyFiles,
 		},
+		{
+			use:     "etcd",


Should we make this more verbose? e.g. etcd-serving-cert

I agree we could make this clearer since there are so many certs.
I'll try etcd-server -- WDYT?

updated this to etcd-server

jamiehannaford · 2018-01-12T17:36:17Z

cmd/kubeadm/app/cmd/phases/certs.go

+			cmdFunc: certsphase.CreateEtcdCertAndKeyFiles,
+		},
+		{
+			use:     "etcd-peer",


etcd-peer-cert?

This is already nested under the certs phase. (kubeadm alpha phase certs etcd-peer)

jamiehannaford · 2018-01-12T17:36:29Z

cmd/kubeadm/app/cmd/phases/certs.go

+			cmdFunc: certsphase.CreateEtcdPeerCertAndKeyFiles,
+		},
+		{
+			use:     "apiserver-etcd-client",


etcd-client-cert?

I made this match phase certs apiserver-kubelet-client.
This naming emphasizes it's an apiserver cert.

Do we want to flip this around?
We can but it won't match the other example of this relationship.
Phases are alpha, so we can also change the apiserver-kubelet-client cert command.

jamiehannaford · 2018-01-12T17:37:56Z

cmd/kubeadm/app/phases/certs/certs.go

+	)
+}
+
+// CreateEtcdCertAndKeyFiles create a new certificate and key file for etcd.


I can create another PR to clean all of these up -- I'd prefer the docstrings to be uniform

jamiehannaford · 2018-01-12T17:38:26Z

cmd/kubeadm/app/phases/certs/certs.go

+	}
+
+	return writeCertificateFilesIfNotExist(
+		cfg.CertificatesDir,


I think I prefer nesting all etcd certs into a new dir, e.g. cfg.CertificatesDir + "/etcd"

I'll take a look at this -- need to consider path joining and tests

730e91c3ae addresses this by hardcoding etcd/ as a prefix in kubeadmconstants.
I found this to be the cleanest implementation.
We're already hardcoding forward-slashes for defaults such as /etc/kubernetes/pki.

I had concerns about adding a subdirectory prefixed to a fileName, but it turns out to be fine:

certutils.WriteKey() does a MkdirAll(filepath.Dir(joinedPath), ...)

testutils.AssertFileExists() does a filepath.Join(dirName, fileName)

Not hardcoding this value requires us to add filepath.Join(cfg.CerticatesDir, kubeadmconstants.Etcd, ... multiple times in 4 files:

diff --git a/cmd/kubeadm/app/phases/certs/certs.go b/cmd/kubeadm/app/phases/certs/certs.go index 86583f4cf5..c377a9e2b1 100644 --- a/cmd/kubeadm/app/phases/certs/certs.go +++ b/cmd/kubeadm/app/phases/certs/certs.go @@ -138,7 +138,7 @@ func CreateEtcdServerCertAndKeyFiles(cfg *kubeadmapi.MasterConfiguration) error } return writeCertificateFilesIfNotExist( - cfg.CertificatesDir, + filepath.Join(cfg.CertificatesDir, kubeadmconstants.Etcd), kubeadmconstants.EtcdServerCertAndKeyBaseName, caCert, etcdServerCert, @@ -162,7 +162,7 @@ func CreateEtcdPeerCertAndKeyFiles(cfg *kubeadmapi.MasterConfiguration) error { } return writeCertificateFilesIfNotExist( - cfg.CertificatesDir, + filepath.Join(cfg.CertificatesDir, kubeadmconstants.Etcd), kubeadmconstants.EtcdPeerCertAndKeyBaseName, caCert, etcdPeerCert, diff --git a/cmd/kubeadm/app/phases/certs/certs_test.go b/cmd/kubeadm/app/phases/certs/certs_test.go index fbd77208ad..343c8c61f8 100644 --- a/cmd/kubeadm/app/phases/certs/certs_test.go +++ b/cmd/kubeadm/app/phases/certs/certs_test.go @@ -579,6 +579,11 @@ func TestCreateCertificateFilesMethods(t *testing.T) { kubeadmconstants.CACertName, kubeadmconstants.CAKeyName, kubeadmconstants.APIServerCertName, kubeadmconstants.APIServerKeyName, kubeadmconstants.APIServerKubeletClientCertName, kubeadmconstants.APIServerKubeletClientKeyName, + filepath.Join(kubeadmconstants.Etcd, kubeadmconstants.EtcdServerCertName), + filepath.Join(kubeadmconstants.Etcd, kubeadmconstants.EtcdServerKeyName), + filepath.Join(kubeadmconstants.Etcd, kubeadmconstants.EtcdPeerCertName), + filepath.Join(kubeadmconstants.Etcd, kubeadmconstants.EtcdPeerKeyName), + kubeadmconstants.APIServerEtcdClientCertName, kubeadmconstants.APIServerEtcdClientKeyName, kubeadmconstants.ServiceAccountPrivateKeyName, kubeadmconstants.ServiceAccountPublicKeyName, kubeadmconstants.FrontProxyCACertName, kubeadmconstants.FrontProxyCAKeyName, kubeadmconstants.FrontProxyClientCertName, kubeadmconstants.FrontProxyClientKeyName, @@ -599,14 +604,20 @@ func TestCreateCertificateFilesMethods(t *testing.T) { expectedFiles: []string{kubeadmconstants.APIServerKubeletClientCertName, kubeadmconstants.APIServerKubeletClientKeyName}, }, { - setupFunc: CreateCACertAndKeyfiles, - createFunc: CreateEtcdServerCertAndKeyFiles, - expectedFiles: []string{kubeadmconstants.EtcdServerCertName, kubeadmconstants.EtcdServerKeyName}, + setupFunc: CreateCACertAndKeyfiles, + createFunc: CreateEtcdServerCertAndKeyFiles, + expectedFiles: []string{ + filepath.Join(kubeadmconstants.Etcd, kubeadmconstants.EtcdServerCertName), + filepath.Join(kubeadmconstants.Etcd, kubeadmconstants.EtcdServerKeyName), + }, }, { - setupFunc: CreateCACertAndKeyfiles, - createFunc: CreateEtcdPeerCertAndKeyFiles, - expectedFiles: []string{kubeadmconstants.EtcdPeerCertName, kubeadmconstants.EtcdPeerKeyName}, + setupFunc: CreateCACertAndKeyfiles, + createFunc: CreateEtcdPeerCertAndKeyFiles, + expectedFiles: []string{ + filepath.Join(kubeadmconstants.Etcd, kubeadmconstants.EtcdPeerCertName), + filepath.Join(kubeadmconstants.Etcd, kubeadmconstants.EtcdPeerKeyName), + }, }, { setupFunc: CreateCACertAndKeyfiles, diff --git a/cmd/kubeadm/app/phases/certs/doc.go b/cmd/kubeadm/app/phases/certs/doc.go index 7e66cd344c..641907e6c8 100644 --- a/cmd/kubeadm/app/phases/certs/doc.go +++ b/cmd/kubeadm/app/phases/certs/doc.go @@ -38,12 +38,12 @@ package certs - apiserver.key - apiserver-kubelet-client.crt - apiserver-kubelet-client.key - - etcd-server.crt - - etcd-server.key - - etcd-peer.crt - - etcd-peer.key - apiserver-etcd-client.crt - apiserver-etcd-client.key + - etcd/server.crt + - etcd/server.key + - etcd/peer.crt + - etcd/peer.key - sa.pub - sa.key - front-proxy-ca.crt diff --git a/cmd/kubeadm/app/phases/etcd/local.go b/cmd/kubeadm/app/phases/etcd/local.go index d25503521f..4a78c8b7c2 100644 --- a/cmd/kubeadm/app/phases/etcd/local.go +++ b/cmd/kubeadm/app/phases/etcd/local.go @@ -75,12 +75,12 @@ func getEtcdCommand(cfg *kubeadmapi.MasterConfiguration) []string { "listen-client-urls": "https://127.0.0.1:2379", "advertise-client-urls": "https://127.0.0.1:2379", "data-dir": cfg.Etcd.DataDir, - "cert-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.EtcdServerCertName), - "key-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.EtcdServerKeyName), + "cert-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.Etcd, kubeadmconstants.EtcdServerCertName), + "key-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.Etcd, kubeadmconstants.EtcdServerKeyName), "trusted-ca-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.CACertName), "client-cert-auth": "true", - "peer-cert-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.EtcdPeerCertName), - "peer-key-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.EtcdPeerKeyName), + "peer-cert-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.Etcd, kubeadmconstants.EtcdPeerCertName), + "peer-key-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.Etcd, kubeadmconstants.EtcdPeerKeyName), "peer-trusted-ca-file": filepath.Join(cfg.CertificatesDir, kubeadmconstants.CACertName), "peer-client-cert-auth": "true", } diff --git a/cmd/kubeadm/app/phases/etcd/local_test.go b/cmd/kubeadm/app/phases/etcd/local_test.go index 920680e059..425a96416c 100644 --- a/cmd/kubeadm/app/phases/etcd/local_test.go +++ b/cmd/kubeadm/app/phases/etcd/local_test.go @@ -82,12 +82,12 @@ func TestGetEtcdCommand(t *testing.T) { "--listen-client-urls=https://127.0.0.1:2379", "--advertise-client-urls=https://127.0.0.1:2379", "--data-dir=/var/lib/etcd", - "--cert-file=etcd-server.crt", - "--key-file=etcd-server.key", + "--cert-file=etcd/server.crt", + "--key-file=etcd/server.key", "--trusted-ca-file=ca.crt", "--client-cert-auth=true", - "--peer-cert-file=etcd-peer.crt", - "--peer-key-file=etcd-peer.key", + "--peer-cert-file=etcd/peer.crt", + "--peer-key-file=etcd/peer.key", "--peer-trusted-ca-file=ca.crt", "--peer-client-cert-auth=true", }, @@ -107,12 +107,12 @@ func TestGetEtcdCommand(t *testing.T) { "--listen-client-urls=https://10.0.1.10:2379", "--advertise-client-urls=https://10.0.1.10:2379", "--data-dir=/var/lib/etcd", - "--cert-file=etcd-server.crt", - "--key-file=etcd-server.key", + "--cert-file=etcd/server.crt", + "--key-file=etcd/server.key", "--trusted-ca-file=ca.crt", "--client-cert-auth=true", - "--peer-cert-file=etcd-peer.crt", - "--peer-key-file=etcd-peer.key", + "--peer-cert-file=etcd/peer.crt", + "--peer-key-file=etcd/peer.key", "--peer-trusted-ca-file=ca.crt", "--peer-client-cert-auth=true", }, @@ -126,12 +126,12 @@ func TestGetEtcdCommand(t *testing.T) { "--listen-client-urls=https://127.0.0.1:2379", "--advertise-client-urls=https://127.0.0.1:2379", "--data-dir=/etc/foo", - "--cert-file=etcd-server.crt", - "--key-file=etcd-server.key", + "--cert-file=etcd/server.crt", + "--key-file=etcd/server.key", "--trusted-ca-file=ca.crt", "--client-cert-auth=true", - "--peer-cert-file=etcd-peer.crt", - "--peer-key-file=etcd-peer.key", + "--peer-cert-file=etcd/peer.crt", + "--peer-key-file=etcd/peer.key", "--peer-trusted-ca-file=ca.crt", "--peer-client-cert-auth=true", },

jamiehannaford · 2018-01-12T18:11:40Z

cmd/kubeadm/app/phases/certs/certs.go

@@ -545,6 +678,63 @@ func getAltNames(cfg *kubeadmapi.MasterConfiguration) (*certutil.AltNames, error
 			altNames.IPs = append(altNames.IPs, ip)
 		} else if len(validation.IsDNS1123Subdomain(altname)) == 0 {
 			altNames.DNSNames = append(altNames.DNSNames, altname)
+		} else {
+			fmt.Printf("[certificates] WARNING: '%s' was not added to the '%s' SAN, because it is not a valid IP or RFC-1123 compliant DNS entry\n", altname, kubeadmconstants.APIServerCertName)


nit: might be worth splitting over several lines

I'll add line-breaks between the args -- I think this is what you mean 👍

also done in fcf0fefda4 -- thanks :)

jamiehannaford · 2018-01-12T18:12:14Z

cmd/kubeadm/app/phases/certs/certs.go

+		} else if len(validation.IsDNS1123Subdomain(altname)) == 0 {
+			altNames.DNSNames = append(altNames.DNSNames, altname)
+		} else {
+			fmt.Printf("[certificates] WARNING: '%s' was not added to the '%s' SAN, because it is not a valid IP or RFC-1123 compliant DNS entry\n", altname, kubeadmconstants.EtcdCertName)


let's define this error once

thanks for the suggestion 👍 this is done (fcf0fefda4)

jamiehannaford · 2018-01-12T18:12:28Z

cmd/kubeadm/app/phases/certs/certs.go

+	}
+
+	// adds additional SAN
+	for _, altname := range cfg.EtcdCertSANs {


can we re-use this logic?

implemented this as appendSANsToAltNames() in certs/pkiutil (fcf0fefda4)

timothysc

Address other minor changes, squash commits, then lgtm.

I debate if we want to shift-out etcd logic if we are going to create mgmt code. /cc @jamiehannaford

timothysc · 2018-01-17T02:06:35Z

cmd/kubeadm/app/constants/constants.go

@@ -64,6 +64,33 @@ const (
 	// APIServerKubeletClientCertCommonName defines kubelet client certificate common name (CN)
 	APIServerKubeletClientCertCommonName = "kube-apiserver-kubelet-client"

+	// EtcdCertAndKeyBaseName defines etcd's server certificate and key base name


We should follow names outlined here: https://coreos.com/etcd/docs/latest/op-guide/security.html , there are a couple of conflicts.

I'll update the etcd-server cert like @jamiehannaford suggests.

For peers, I see:

this PR names the Peer Cert "etcd-peer.crt"

the CoreOS example names it "member1.crt"

I find this naming strange. It doesn't match the flags: --peer-cert-file, --peer-key-file
The term "member" is used loosely in the docs I've come across.

We have an ansible example in the kubernetes/contrib repo that uses the peer naming:
https://github.com/kubernetes/contrib/blob/master/ansible/roles/etcd/defaults/main.yaml#L21-L31

This CoreOS example mentions member.crt in the description but uses infra0-peer.crt in the code:
https://github.com/coreos/etcd/blob/master/Documentation/op-guide/clustering.md#self-signed-certificates

This other CoreOS example just reuses the server crt as the peer crt:
https://github.com/coreos/etcd/blob/master/hack/tls-setup/Procfile

timothysc · 2018-01-17T02:08:48Z

cmd/kubeadm/app/phases/certs/certs.go

+	}
+
+	return writeCertificateFilesIfNotExist(
+		cfg.CertificatesDir,


luxas

IMO, one year would be ok (it's the default value?)

The default CA validity for the k8s root CA is ten years sigh. The individual API Server certs are then valid for one year.

I debate if we want to shift-out etcd logic if we are going to create mgmt code

I don't think this is lifecycle code. I'm okay with keeping this cert generation cert generation and then including what's needed for etcd here as well. The methods are public so anyone can import anyway.

Good job so far @stealthybox!

luxas · 2018-02-02T16:01:43Z

cmd/kubeadm/app/phases/certs/certs.go

+		AltNames:   *altNames,
+		Usages:     []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+	}
+	etcdCert, etcdKey, err := pkiutil.NewCertAndKey(caCert, caKey, config)


NewCertAndKey is duplicated across all new methods... any chance these methods could just return certutil.Config and then let the caller run this method? Do you think that would make sense?

I can look at this in another PR to refactor all of these methods.
These are basically copy-paste.

Changing the return type would probably warrant a rename and new docstrings, and these are all public.

luxas · 2018-02-02T16:02:35Z

cmd/kubeadm/app/phases/certs/certs.go

+// localhost is included in the SAN since this is the interface the etcd static pod listens on
+// hostname and API.AdvertiseAddress are excluded since etcd does not listen on this interface by default
+// the user can override the listen address with Etcd.ExtraArgs and add SANs with EtcdCertSANs
+func getEtcdAltNames(cfg *kubeadmapi.MasterConfiguration) (*certutil.AltNames, error) {


this logic could make sense in an other package, like pkiutil or the general util instead of here in a phase package.

I like this suggestion.
It touches code that's outside of the scope of making this patch function.
Let's do a follow-up PR

@luxas -- I ended up implementing this while considering @jamiehannaford's comments.
See fcf0fefda4.

The one part I don't like about it is that I've added user-facing warnings to the pkiutil package.
Lmk what you think?

mattkelly · 2018-02-21T18:30:55Z

cmd/kubeadm/app/cmd/phases/certs.go

@@ -60,8 +60,8 @@ var (
 	apiServerCertLongDesc = fmt.Sprintf(normalizer.LongDesc(`
 		Generates the API server serving certificate and key and saves them into %s and %s files.

-		The certificate includes default subject alternative names and additional sans eventually provided by the user;
-		default sans are: <node-name>, <apiserver-advertise-address>, kubernetes, kubernetes.default, kubernetes.default.svc, 
+		The certificate includes default subject alternative names and additional SANs provided by the user;


Maybe capitalize Subject Alternative Names so that it's more clear that SAN (used later) is abbreviating that (applies below too).

mattkelly · 2018-02-21T18:34:43Z

cmd/kubeadm/app/phases/certs/certs.go

+	)
+}
+
+// CreateEtcdServerCertAndKeyFiles create a new certificate and key file for etcd.


creates (applies for other function comments below)

mattkelly · 2018-02-21T18:35:28Z

cmd/kubeadm/app/phases/certs/certs.go

+}
+
+// CreateEtcdServerCertAndKeyFiles create a new certificate and key file for etcd.
+// If the etcd serving certificate and key file already exist in the target folder, they are used only if evaluated equal; otherwise an error is returned.


Maybe evaluated to be equal (applies for other function comments below)

mattkelly · 2018-02-21T18:40:32Z

cmd/kubeadm/app/phases/certs/certs.go

+	}
+	etcdServerCert, etcdServerKey, err := pkiutil.NewCertAndKey(caCert, caKey, config)
+	if err != nil {
+		return nil, nil, fmt.Errorf("failure while creating etcd key and certificate: %v", err)


Maybe be more specific, etcd server key

mattkelly · 2018-02-21T18:41:17Z

cmd/kubeadm/app/phases/certs/certs_test.go

+
+	etcdServerCert, _, err := NewEtcdServerCertAndKey(cfg, caCert, caKey)
+	if err != nil {
+		t.Fatalf("failed creation of cert and key: %v", err)


More specific: of etcd server cert and key

mattkelly · 2018-02-21T18:42:38Z

cmd/kubeadm/app/phases/certs/pkiutil/pki_helpers.go

@@ -246,3 +251,101 @@ func pathForKey(pkiPath, name string) string {
 func pathForPublicKey(pkiPath, name string) string {
 	return filepath.Join(pkiPath, fmt.Sprintf("%s.pub", name))
 }
+
+// GetAPIServerAltNames builds an AltNames object for to be used when generating apiserver certificate


Delete for: AltNames object to be used...

mattkelly · 2018-02-21T18:46:14Z

cmd/kubeadm/app/phases/certs/pkiutil/pki_helpers.go

+}
+
+// GetEtcdAltNames builds an AltNames object for generating the etcd server certificate
+// localhost is included in the SAN since this is the interface the etcd static pod listens on


I don't think godoc will break this up into multiple lines when the docs are generated, so this may read as one giant run-on sentence. Consider breaking it up into actual sentences.

Thanks for catching this one @mattkelly!
It's now fixed in the most recent rebase.

I have another PR ready to address the other typos over the whole codebase -- I'll make sure to check your other comments when we get that submitted.

ericchiang · 2018-02-22T01:09:49Z

After todays discussion, I'm not certain if we should plumb through options for a separate CA for etcd, but it seems prudent.

This is what the CIS benchmark recommends. If not, you have to use the --peer-cert-allowed-cn to restrict the set of client certs that can access API.

https://coreos.com/etcd/docs/latest/op-guide/security.html#notes-for-tls-authentication

stealthybox · 2018-02-22T17:57:16Z

@ericchiang Thanks -- Do you think we should implement the etcd CA then?
You mentioned there are some limitations with this regarding some CSR API in etcd-io/etcd#8262.

--peer-cert-allowed-cn is a new v3.3+ feature so that really complicates things.

Am I understanding that we would also have to enable etcd's RBAC and make a user for the apiserver so that we could prevent unauthorized clients from connecting?
(--client-cert-auth=true && etcdctl auth enable)
Would we just use the root etcd user or role?

We would also need to make sure that etcd-peer certs aren't permitted to use Kubernetes API's.

ericchiang · 2018-02-22T18:40:51Z

@ericchiang Thanks -- Do you think we should implement the etcd CA then?
You mentioned there are some limitations with this regarding some CSR API in etcd-io/etcd#8262.

Separate CAs make the policy simpler, but complicates the PKI management. I'm not familiar enough with kubeadm HA efforts to say which trade off makes sense.

If you don't have a good enough reason to centralize it (e.g. you're using a CSR API that doesn't support intermediate CAs), separate CAs probably makes sense.

Am I understanding that we would also have to enable etcd's RBAC and make a user for the apiserver so that we could prevent unauthorized clients from connecting?
(--client-cert-auth=true && etcdctl auth enable)
Would we just use the root etcd user or role?

You can just use the root user, or map the root role onto something more convenient for your X509 CN.

We would also need to make sure that etcd-peer certs aren't permitted to use Kubernetes API's.

Etcd peers can arbitrarily modify etcd data. They have backdoor access to the Kubernetes API already :)

timothysc

/approve

Please squash your commits, open a separate issue to follow on with a separate CA and I'll lgtm this one, b/c code freeze is Monday.

- Generate Server and Peer cert for etcd - Generate Client cert for apiserver - Add flags / hostMounts for etcd static pod - Add flags / hostMounts for apiserver static pod - Generate certs on upgrade of static-pods for etcd/kube-apiserver - Modify logic for appending etcd flags to staticpod to be safer for external etcd

stealthybox · 2018-02-23T23:18:59Z

Down from 20 commits to 4 😅

seriously the most intense rebase I've ever done... I'm exhausted

❯ git branch | grep 594
* feature/kubeadm_594-etcd_tls
  feature/kubeadm_594-etcd_tls--subdir-draft
  feature/kubeadm_594-etcd_tls__rebase
  feature/kubeadm_594-etcd_tls__unsquashed
  feature/kubeadm_594-etcd_tls__unsquashed__2
  feature/kubeadm_594-etcd_tls__unsquashed__2__autogendocs
  feature/kubeadm_594-etcd_tls__unsquashed__3__move_typos_autogendocs
  feature/kubeadm_594-etcd_tls__unsquashed__3__move_typos_autogendocs__squash1
  feature/kubeadm_594-etcd_tls__unsquashed__3__move_typos_autogendocs__squash1__move_typos
  feature/kubeadm_594-etcd_tls__unsquashed__4
  feature/kubeadm_594-etcd_tls__unsquashed__5__veryvery_squashed

I made sure there were no code changes

@jamiehannaford

- Place etcd server and peer certs & keys into pki subdir - Move certs.altName functions to pkiutil + add appendSANstoAltNames() Share the append logic for the getAltName functions as suggested by @jamiehannaford. Move functions/tests to certs/pkiutil as suggested by @luxas. Update Bazel BUILD deps - Warn when an APIServerCertSANs or EtcdCertSANs entry is unusable - Add MasterConfiguration.EtcdPeerCertSANs - Move EtcdServerCertSANs and EtcdPeerCertSANs under MasterConfiguration.Etcd

- Fix typos in tests for upgrade phase - Rename loadCertificateAuthorithy() --> loadCertificateAuthority() - Disambiguate apiKubeletClientCert & apiEtcdClientCert - Parameterize hard-coded certs_test config + log tempCertsDir

stealthybox · 2018-02-24T00:42:36Z

@timothysc I've opened kubernetes/kubeadm#710 for the separate CA.
I can't add labels in that repo -- feel free to triage it to the 1.10 milestone. I'll put up another PR.

timothysc · 2018-02-24T00:45:25Z

/test pull-kubernetes-unit

timothysc

/lgtm

k8s-ci-robot · 2018-02-24T01:24:50Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: stealthybox, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [timothysc]
docs/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-github-robot · 2018-02-25T05:19:40Z

Automatic merge from submit-queue (batch tested with PRs 59159, 60318, 60079, 59371, 57415). If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 19, 2017

k8s-github-robot assigned jamiehannaford and timothysc Dec 19, 2017

k8s-ci-robot assigned luxas Dec 19, 2017

stealthybox force-pushed the feature/kubeadm_594-etcd_tls branch from ef708b6 to baf3a14 Compare December 20, 2017 16:50

k8s-github-robot added the kind/old-docs label Dec 20, 2017

luxas added this to the v1.10 milestone Dec 22, 2017

k8s-ci-robot assigned fabriziopandini and kad Dec 22, 2017

0xmichalis reviewed Dec 28, 2017

View reviewed changes

stealthybox force-pushed the feature/kubeadm_594-etcd_tls branch from 121b135 to ed08db6 Compare December 29, 2017 06:52

xiangpengzhao reviewed Jan 9, 2018

View reviewed changes

k8s-ci-robot requested a review from mattmoyer January 9, 2018 15:57

jamiehannaford suggested changes Jan 12, 2018

View reviewed changes

timothysc suggested changes Jan 17, 2018

View reviewed changes

luxas reviewed Feb 2, 2018

View reviewed changes

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 7, 2018

stealthybox force-pushed the feature/kubeadm_594-etcd_tls branch from ed08db6 to d2c5677 Compare February 14, 2018 06:44

timothysc added the sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. label Feb 20, 2018

stealthybox force-pushed the feature/kubeadm_594-etcd_tls branch from eef1d38 to 4407fd9 Compare February 21, 2018 17:13

mattkelly reviewed Feb 21, 2018

View reviewed changes

timothysc approved these changes Feb 21, 2018

View reviewed changes

stealthybox force-pushed the feature/kubeadm_594-etcd_tls branch from 4407fd9 to 530d945 Compare February 23, 2018 17:57

timothysc reviewed Feb 23, 2018

View reviewed changes

timothysc added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 23, 2018

stealthybox force-pushed the feature/kubeadm_594-etcd_tls branch 2 times, most recently from 6c4ae20 to 5db8884 Compare February 23, 2018 23:00

stealthybox force-pushed the feature/kubeadm_594-etcd_tls branch from 5db8884 to 2406f23 Compare February 23, 2018 23:08

stealthybox added 3 commits February 23, 2018 17:05

Fix typos

f61430d

- Fix typos in tests for upgrade phase - Rename loadCertificateAuthorithy() --> loadCertificateAuthority() - Disambiguate apiKubeletClientCert & apiEtcdClientCert - Parameterize hard-coded certs_test config + log tempCertsDir

Update autogenerated docs

509e9af

stealthybox force-pushed the feature/kubeadm_594-etcd_tls branch from 2406f23 to 509e9af Compare February 24, 2018 00:05

stealthybox mentioned this pull request Feb 24, 2018

Add separate CA for etcd certificates kubernetes/kubeadm#710

Closed

timothysc approved these changes Feb 24, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 24, 2018

k8s-github-robot merged commit b32e9c4 into kubernetes:master Feb 25, 2018

jessicaochen mentioned this pull request Mar 1, 2018

[job failure] periodic-kubernetes-e2e-kubeadm-gce-selfhosting #59766

Closed

stealthybox mentioned this pull request Mar 3, 2018

etcd fails kubelet's health checks kubernetes/kubeadm#720

Closed

stealthybox mentioned this pull request Mar 18, 2018

kubeadm: etcd certs missing in self-hosted deploy #61322

Closed

	certAndKeyDir := kubeadmapiext.DefaultCertificatesDir
	shouldBackup, err := shouldBackupAPIServerCertAndKey(certAndKeyDir, newK8sVer)
	// Don't fail the upgrade phase if failing to determine to backup kube-apiserver cert and key.
	if err != nil {
	fmt.Printf("[postupgrade] WARNING: failed to determine to backup kube-apiserver cert and key: %v", err)
	} else if shouldBackup {
	// Don't fail the upgrade phase if failing to backup kube-apiserver cert and key.
	if err := backupAPIServerCertAndKey(certAndKeyDir); err != nil {
	fmt.Printf("[postupgrade] WARNING: failed to backup kube-apiserver cert and key: %v", err)
	}
	if err := certsphase.CreateAPIServerCertAndKeyFiles(cfg); err != nil {
	errs = append(errs, err)
	}
	}

Feature/kubeadm 594 etcd TLS on init/upgrade #57415

Feature/kubeadm 594 etcd TLS on init/upgrade #57415

Conversation

stealthybox commented Dec 19, 2017 • edited Loading

on the master

stealthybox commented Dec 19, 2017

luxas commented Dec 22, 2017

Choose a reason for hiding this comment

stealthybox Dec 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stealthybox commented Dec 29, 2017

xiangpengzhao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stealthybox Feb 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc commented Jan 9, 2018

xiangpengzhao commented Jan 9, 2018

stealthybox commented Jan 9, 2018

timothysc commented Jan 9, 2018

xiangpengzhao commented Jan 12, 2018

jamiehannaford left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stealthybox Feb 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stealthybox Feb 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stealthybox Feb 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stealthybox Feb 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stealthybox Feb 19, 2018 • edited Loading

Choose a reason for hiding this comment

timothysc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stealthybox Feb 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luxas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stealthybox Feb 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stealthybox commented Dec 19, 2017 •

edited

Loading

stealthybox Dec 29, 2017 •

edited

Loading

stealthybox Feb 19, 2018 •

edited

Loading

stealthybox Feb 14, 2018 •

edited

Loading

stealthybox Feb 14, 2018 •

edited

Loading

stealthybox Feb 19, 2018 •

edited

Loading

stealthybox Feb 19, 2018 •

edited

Loading

stealthybox Feb 19, 2018 •

edited

Loading

stealthybox Feb 14, 2018 •

edited

Loading

stealthybox Feb 19, 2018 •

edited

Loading

ericchiang commented Feb 22, 2018 •

edited

Loading