Investigate if Liveness Probing is Production Grade

[johnrk-zz]

During Monday's (7/13) demo of the Operator, it was identified that our liveness probing for Kubernetes may need to improve. This will be helpful to have to have proper end to end testing.

For our end to end testing, we verify that clusters are setup properly. But we don't check to see if they're running properly. We don't run any SQL queries, and we don't check for liveness.

[vladdy]

Just a small correction. e2e tests check for liveness of all pods - https://github.com/cockroachdb/cockroach-operator/blob/master/e2e/assert.go#L138

[chrislovecnm]

@chrisseto can we close this?

[chrisseto]

I'm tempted to say that we should leave in this open? In CC, we actually don't have liveness probes installed. We've found that the probes timeout at high loads, so k8s kills the pods making the issue even worse.
We've not encountered any issues that would be solved by adding it back in however.

[udnay]

Can we close this?

[mbrancato]

Noting here that the upstream manifests have disabled liveness probes and we experienced liveness probe failures under heavy load causing more latency in the cluster.

Support referenced these:
cockroachdb/cockroach#67080
cockroachdb/cockroach#44832