Add livez and readyz for etcd

[siyuanfoundation]

#16007

This is a prototype for adding livez/readyz support to etcd (design doc).
This pr is setting up the general structure for livez/ready checks, with only 2 simple checks implemented.

[siyuanfoundation]

Tested with local server
curl '127.0.0.1:2379/readyz?verbose'
output:

[+]ping ok
[+]serializable_read ok
[+]data_corruption ok
[+]defrag_active ok
readyz check passed

curl '127.0.0.1:2379/livez?verbose'
output:

[+]ping ok
[+]serializable_read ok
livez check passed

[tjungblu]

Cool stuff and thanks for the PR, I was wondering why you're adding this to the GRPC API? We already have a health handler on the http server: https://github.com/etcd-io/etcd/blob/main/server/embed/etcd.go#L746

Just FYI, we had huge issues in the past with running etcdctl commands as a health probes in openshift with regards to zombie processes. We're much happier with just letting kubelet hit an http endpoint instead.

[chaochn47]

We already have a health handler on the http server: https://github.com/etcd-io/etcd/blob/main/server/embed/etcd.go#L746

I think we can start from http endpoint. I don’t think backporting gRPC changes to release-3.5 is feasible and accepted. It will literally change the customer facing API even if it is just maintenance service.

[siyuanfoundation]

We already have a health handler on the http server: https://github.com/etcd-io/etcd/blob/main/server/embed/etcd.go#L746

I think we can start from http endpoint. I don’t think backporting gRPC changes to release-3.5 is feasible and accepted. It will literally change the customer facing API even if it is just maintenance service.

Removed gPRC change.

[wenjiaswe]

cc @ahrtr @mitake @jmhbnz @ptabor @lavacat @fuweid

[fuweid]

LGTM

Nice work!

[ahrtr]

Please rebase this PR and resolve the left comment, thx

[serathius]

Any reason to implement verbose for per check endpoint?

[siyuanfoundation]

per check endpoint still print the detailed error message in verbose, and no details in non verbose. In theory, the user can get all the details from the root path verbose. But I think it still makes sense to follow the same paradigm.

[siyuanfoundation]

@ahrtr Rebased the PR. Please review. Thanks!

[logicalhan]

/lgtm

[ahrtr]

LGTM

Thanks for your first contribution, and great work!

Please also add a followup item to update the doc in etcd-io/website.

Question about serializable read

[serathius]

Please note that this PR doesn't implement the full design as it was presented. Livez works correct, however we are missing checks for readyz making it not very useful. Please see 80ab2ad

I think this is ok to merge, however until we finish implementing readyz, we shouldn't backport nor document the new endpoints.