Rolling Update Usually results in etcd/api-server Related Downtime

[LeeHampton]

1. What kops version are you running? The command kops version, will display
this information.

Version 1.17.0

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

1.17.6

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

kops rolling-update cluster ${CLUSTER_NAME} --yes --force --instance-group-roles Master

5. What happened after the commands executed?

This is an HA 3-master cluster on AWS using m5.2xlarge instances with attached gp2 EBS for the etcd disks. I'm using version 3.4.3 for etcd with etcd-manager.

This isn't fully deterministic, but basically after a master gets taken down in a rolling update I often get some variation of the following.

The API Server logs some etcd related errors -- either a leader election error or a timeout error:

E0629 18:45:05.248843       1 status.go:71] apiserver received an error that is not an metav1.Status : rpctypes.EtcdError{code:0xe, desc:"etcdserver: request timed out"}

or

apiserver received an error that is not an metav1.Status rpctypes.EtcdError{code:0xe, desc:"etcdserver: leader changed"}

Often this causes the API server to restart in an unhealthy state:

healthz check failed
[-]etcd failed: reason withheld

But this seems to resolve quickly and the API server gets back to normal. However, during this time, services that rely on the kube API server get timeout errors when trying to connect to it.

Over on the etcd side, I see the usual logs around a peer member not being reachable, etc. during the time while the new peer is unavailable. However, I also see some timeout error messages in the etcd logs:

2020-06-30 16:25:38.979648 W | etcdserver: server is likely overloaded
2020-06-30 16:25:38.979653 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 68.908672ms, to 607a5729c01d3c7d)

And things like this:

etcdserver: read-only range request "key:\"/registry/configmaps/cert-manager/cert-manager-cainjector-leader-election-core\" " with result "range_response_count:1 size:523" took too long (5.896109403s) to execute

These, too, resolve after a few minutes and don't seem to come back.

It seems like any time a new etcd leader election goes down, we're guaranteed at least 30 seconds of downtime and some weird bootup issues. I'm not sure what to do here. A 3 node etcd cluster should survive downtime of 1 of its members, but currently if 1 of the nodes goes down, basically k8s becomes fully unavailable. This is a problem for our cluster because it relies on the k8s api server for a lot of functionality.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.