Update to etcd-manager 3.0.20200429

[justinsb]

Contains the workaround for 1-year certificate expiry.

3.0.2020428 changes

• Release notes for 3.0.20200307 #303
• Add support for etcd 3.3.17 #304
• Adding client usage extension for server cert (DNS is a SPOF; make sure there are replicas #305) #306
• Add a check to renew certificates on startup if they expire in 60 days or less #309
• Try github actions #310
• Upgrade bazel to 2.2.0 #311
• Update to go 1.13.10 #314
• Bazel: update dependency #316
• e2e tests should wait for cluster readiness #318
• Remove old bazel versions from travis #317
• Always renew certificates #313

3.0.20200429

* Upgrade aws-sdk-go [#320](https://github.com/kopeio/etcd-manager/pull/320)
* Release notes for 3.0.20200428 [#319](https://github.com/kopeio/etcd-manager/pull/319)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justinsb]

When reviewing, please in particular look over the certificate rotation changes:

• Add a check to renew certificates on startup if they expire in 60 days or less #309
• Always renew certificates #313

We might also want to bump to a version that includes kopeio/etcd-manager#320 , cc @rifelpet . Not sure if we would backport though.

[rifelpet]

If you're fine with the extra effort of two etcd-manager releases and version bumps in kops, i'd be fine with keeping them separate and only backporting this PR. (maybe backport the aws-sdk upgrade to kops 1.17 only)

[hakman]

Looks like this is working.

I0428 19:48:47.014841   14400 volumes.go:85] AWS API Request: ec2metadata/GetToken
I0428 19:48:47.016796   14400 volumes.go:85] AWS API Request: ec2metadata/GetDynamicData
I0428 19:48:47.017811   14400 volumes.go:85] AWS API Request: ec2metadata/GetMetadata
I0428 19:48:47.018530   14400 volumes.go:85] AWS API Request: ec2metadata/GetMetadata
I0428 19:48:47.019224   14400 volumes.go:85] AWS API Request: ec2metadata/GetMetadata
I0428 19:48:47.019895   14400 main.go:279] Mounting available etcd volumes matching tags [k8s.io/etcd/main k8s.io/role/master=1 kubernetes.io/cluster/etcd-manager.text.com=owned]; nameTag=k8s.io/etcd/main
I0428 19:48:47.021607   14400 volumes.go:85] AWS API Request: ec2/DescribeVolumes
I0428 19:48:47.287785   14400 mounter.go:72] Master volume "vol-0e60ca2264338a506" is attached at "/dev/xvdv"
I0428 19:48:47.288059   14400 mounter.go:86] Doing safe-format-and-mount of /dev/xvdv to /mnt/master-vol-0e60ca2264338a506
I0428 19:48:47.288153   14400 volumes.go:233] volume vol-0e60ca2264338a506 not mounted at /rootfs/dev/xvdv
I0428 19:48:47.288257   14400 volumes.go:247] found nvme volume "nvme-Amazon_Elastic_Block_Store_vol0e60ca2264338a506" at "/dev/nvme1n1"
I0428 19:48:47.288330   14400 mounter.go:125] Found volume "vol-0e60ca2264338a506" mounted at device "/dev/nvme1n1"
I0428 19:48:47.289162   14400 mounter.go:179] Device already mounted on "/mnt/master-vol-0e60ca2264338a506", verifying it is our device
I0428 19:48:47.289339   14400 mounter.go:191] Found existing mount of "/dev/nvme1n1" at "/mnt/master-vol-0e60ca2264338a506"
I0428 19:48:47.289560   14400 mount_linux.go:153] Detected OS without systemd
I0428 19:48:47.290513   14400 mounter.go:232] matched device "/dev/nvme1n1" and "/dev/nvme1n1" via '\x00'
I0428 19:48:47.290598   14400 mounter.go:94] mounted master volume "vol-0e60ca2264338a506" on /mnt/master-vol-0e60ca2264338a506
I0428 19:48:47.290659   14400 main.go:294] discovered IP address: 10.4.211.27
I0428 19:48:47.290675   14400 main.go:299] Setting data dir to /rootfs/mnt/master-vol-0e60ca2264338a506
I0428 19:48:47.292286   14400 certs.go:106] existing certificate not valid after 2021-04-27T01:03:31Z; will regenerate
I0428 19:48:47.292372   14400 certs.go:167] generating certificate for "etcd-manager-server-etcd-a"
I0428 19:48:47.295333   14400 certs.go:106] existing certificate not valid after 2021-04-27T01:03:31Z; will regenerate
I0428 19:48:47.295412   14400 certs.go:167] generating certificate for "etcd-manager-client-etcd-a"
I0428 19:48:47.298596   14400 server.go:71] starting GRPC server using TLS, ServerName="etcd-manager-server-etcd-a"
I0428 19:48:47.299333   14400 etcdserver.go:534] starting etcd with state cluster:<cluster_token:"OcMpF_NV6lgp-L9yfehlNg" nodes:<name:"etcd-a" peer_urls:"https://etcd-a.internal.etcd-manager.text.com:2380" client_urls:"https://etcd-a.internal.etcd-manager.text.com:4001" quarantined_client_urls:"https://etcd-a.internal.etcd-manager.text.com:3994" tls_enabled:true > > etcd_version:"3.4.3" 
I0428 19:48:47.299593   14400 etcdserver.go:543] starting etcd with datadir /rootfs/mnt/master-vol-0e60ca2264338a506/data/OcMpF_NV6lgp-L9yfehlNg
I0428 19:48:47.300614   14400 pki.go:39] generating peer keypair for etcd: {CommonName:etcd-a Organization:[] AltNames:{DNSNames:[etcd-a.internal.etcd-manager.text.com] IPs:[127.0.0.1]} Usages:[2 1]}
I0428 19:48:47.302601   14400 certs.go:106] existing certificate not valid after 2021-04-27T01:03:44Z; will regenerate
I0428 19:48:47.302711   14400 certs.go:167] generating certificate for "etcd-a"
I0428 19:48:47.316049   14400 pki.go:79] building client-serving certificate: {CommonName:etcd-a Organization:[] AltNames:{DNSNames:[etcd-a.internal.etcd-manager.text.com etcd-a.internal.etcd-manager.text.com] IPs:[127.0.0.1 127.0.0.1]} Usages:[1 2]}
I0428 19:48:47.317621   14400 certs.go:106] existing certificate not valid after 2021-04-27T01:03:44Z; will regenerate
I0428 19:48:47.317694   14400 certs.go:167] generating certificate for "etcd-a"
I0428 19:48:47.576628   14400 certs.go:167] generating certificate for "etcd-a"
I0428 19:48:47.580350   14400 etcdprocess.go:180] executing command /opt/etcd-v3.4.3-linux-amd64/etcd [/opt/etcd-v3.4.3-linux-amd64/etcd]
2020-04-28 19:48:47.599552 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://etcd-a.internal.etcd-manager.text.com:4001
2020-04-28 19:48:47.599602 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/rootfs/mnt/master-vol-0e60ca2264338a506/pki/OcMpF_NV6lgp-L9yfehlNg/clients/server.crt
2020-04-28 19:48:47.599669 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
2020-04-28 19:48:47.599686 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/rootfs/mnt/master-vol-0e60ca2264338a506/data/OcMpF_NV6lgp-L9yfehlNg
2020-04-28 19:48:47.599709 I | pkg/flags: recognized and used environment variable ETCD_ENABLE_V2=false
2020-04-28 19:48:47.599737 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://etcd-a.internal.etcd-manager.text.com:2380
2020-04-28 19:48:47.599743 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-a=https://etcd-a.internal.etcd-manager.text.com:2380
2020-04-28 19:48:47.599748 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
2020-04-28 19:48:47.599759 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=OcMpF_NV6lgp-L9yfehlNg
2020-04-28 19:48:47.599765 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/rootfs/mnt/master-vol-0e60ca2264338a506/pki/OcMpF_NV6lgp-L9yfehlNg/clients/server.key
2020-04-28 19:48:47.599879 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:4001
2020-04-28 19:48:47.599929 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
2020-04-28 19:48:47.599961 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-a
2020-04-28 19:48:47.600036 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/rootfs/mnt/master-vol-0e60ca2264338a506/pki/OcMpF_NV6lgp-L9yfehlNg/peers/me.crt
2020-04-28 19:48:47.600049 I | pkg/flags: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
2020-04-28 19:48:47.600131 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/rootfs/mnt/master-vol-0e60ca2264338a506/pki/OcMpF_NV6lgp-L9yfehlNg/peers/me.key
2020-04-28 19:48:47.600220 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/rootfs/mnt/master-vol-0e60ca2264338a506/pki/OcMpF_NV6lgp-L9yfehlNg/peers/ca.crt
2020-04-28 19:48:47.600245 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/rootfs/mnt/master-vol-0e60ca2264338a506/pki/OcMpF_NV6lgp-L9yfehlNg/clients/ca.crt
2020-04-28 19:48:47.600254 W | pkg/flags: unrecognized environment variable ETCD_LISTEN_METRICS_URLS=
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2020-04-28 19:48:47.600359 I | etcdmain: etcd Version: 3.4.3
2020-04-28 19:48:47.600364 I | etcdmain: Git SHA: 3cf2f69b5
2020-04-28 19:48:47.600368 I | etcdmain: Go Version: go1.12.12
2020-04-28 19:48:47.600371 I | etcdmain: Go OS/Arch: linux/amd64
2020-04-28 19:48:47.600375 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2020-04-28 19:48:47.601142 N | etcdmain: the server is already initialized as member before, starting as etcd member...
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead

[justinsb]

Updated to 3.0.20200429, including the new AWS regions, just for simplicity :-)

[rdrgmnzs]

@justinsb do you want to add the new etcd version to https://github.com/kubernetes/kops/blob/master/pkg/model/components/etcdmanager/options.go#L82 in this PR or a separate one?

[justinsb]

I don't think there's a missing version there @rdrgmnzs ?

kops/pkg/model/components/etcdmanager/options.go

Line 82 in e39ea8d

var supportedEtcdVersions = []string{"2.2.1", "3.1.12", "3.2.18", "3.2.24", "3.3.10", "3.3.13", "3.4.3"}

vs

https://github.com/kopeio/etcd-manager/blob/84bd90884129b71ff5aae0aa36e88ab196dd2af5/pkg/etcdversions/mappings.go#L24-L33

[justinsb]

Oops - now I see it! I'll add 3.3.17 :-)

[justinsb]

Added 3.3.17 :-)

[hakman]

@justinsb @rifelpet anything left to move this along?

[rdrgmnzs]

This lgtm FYI, was testing internally and it's working as intended.

[rifelpet]

I havent had a chance to test it, but if everyone else has had success then I think that's sufficient :)

[justinsb]

I'd like to merge and get this into our test grid, then start the backports...

We should also write up an advisory doc, as if it was a security issue. Has anyone started on one?

[rifelpet]

I have not started one, feel free to open a PR with a draft and we can iterate on it, or I can later tonight.

[hakman]

@justinsb should we also do a quick release of 1.18 alpha 3 in case there are some people eager to test it in the community?

[hakman]

/lgtm

[hakman]

HA tests seem to be going pretty well also:
https://testgrid.k8s.io/sig-cluster-lifecycle-kops#kops-aws-ha-euwest1