etcd pod health/liveliness probe fails if auth is enabled

[hasbro17]

The current probe used for determining etcd liveliness/health is a linearizable get.

etcd-operator/pkg/util/k8sutil/pod_util.go

Lines 72 to 73 in 3ceb27a

	// etcd pod is alive only if a linearizable get succeeds.
	cmd := "ETCDCTL_API=3 etcdctl get foo"

This ensures that the etcd pod is ready to participate in consensus and serve requests.

After auth is enabled on a deployed etcd cluster the etcd-pod's probe will fail since the request is not authenticated.

One solution is to replace the linearizable get with endpoints health.
https://github.com/coreos/etcd/tree/master/etcdctl#endpoint-health

For this to work we need to ensure that endpoints health would give the same guarantees as a linearizable get, and that it works on an auth enabled cluster.

[stevanmilic]

Hey @hasbro17 is there any temporary solution to this problem? i.e. what could I do to make the command use endpoints health in the pod configuration? since manually editing it can't work because of permissions (and besides you don't wanna do that).

[avorima]

@stevanmilic The endpoints health command does a get 'health' so it's not really a fix. At least on my v1.12.2 k8s with v0.9.3 etcd-operator.

[rafi]

The endpoint health still requires a username. However, endpoint status does not. Why weren't the probes changed to endpoint status ?