Fix statefulset PreSync to work for certain cases of unhealthy etcd clusters upon druid upgrade

[shreyas-s-rao]

How to categorize this PR?

/area quality
/kind bug

What this PR does / why we need it:
This PR fixes statefulset PreSync behavior, to no longer wait for etcd pods to be updated with the latest sts spec, or to be ready. Instead, PreSync will now simply check whether the pods have the expected labels, and continue with the next steps. This fixes certain cases where an etcd cluster is unhealthy due to a wrong reconfiguration of the etcd configmap, causing one of the three etcd pods to fail, which causes the PreSync step to get stuck in the pod updated+ready check, and not allow druid to run Sync which reconciles and potentially fixes the issue by reverting the configmap to the correct state.

Which issue(s) this PR fixes:
Fixes #818

Special notes for your reviewer:
While working on this fix, @unmarshall and I found certain cases where PreSync cannot succeed. Consider the following case:

1. Druid is running with old version (v0.22.0)
2. 3-node etcd cluster is deployed and ready
3. Etcd spec is updated with non-existent secret reference (TLS or backup store)
4. Etcd is reconciled, causing the rollout of the statefulset pods to start, updating pod etcd-2 to go into error state since the mentioned secret does not exist.
   1. This unhealthy pod can be caused by any other reason as well, and not necessarily a misconfiguration. We simply use this as an example to show this case.
5. Upgrade druid to v0.23.0 (or current master version)
6. Reconcile the Etcd. This causes the PreSync to get stuck since the statefulset rollout is stuck since the pod never becomes ready.
7. Druid marks the Etcd status.lastOperation as Operation: Reconcile, State: Failed.
8. Now, the user sees this and figures out the Etcd spec misconfiguration issue, corrects the Etcd spec with correct configuration and reconciles the Etcd again. Druid will now succeed and the etcd cluster becomes healthy and also updated with new labels.
9. Thus, misconfiguration or transient errors can be overcome by the current druid logic after this fix PR.
10. But, if before the druid upgrade, there is a data corruption in the etcd cluster, causing a quorum loss, then the new druid PreSync will not be able to handle this, and this is a current limitation of the reconciliation logic today. Druid will still mark the Reconcile operation as Failed, and an operator will need to manually look into it and fix it by performing a recovery from quorum loss using this guide.

/invite @acumino

Release note:

Before upgrading druid to `v0.23.0+`, please ensure that druid is running with at least `v0.22.3+`. This is required to avoid any downtime during the upgrade of the etcds by the new druid version, as well as to ensure backward compatibility of your etcds, in case you wish to downgrade back to `v0.22.3+`.

[gardener-robot]

@shreyas-s-rao Command "/InviteCommand" failed with "Reviews may only be requested from collaborators. One or more of the users or teams you specified is not a collaborator of the gardener/etcd-druid repository.".

Additional Information

Redacted in public. Check backend logs.

[gardener-robot]

@shreyas-s-rao Command "/InviteCommand" failed with "Reviews may only be requested from collaborators. One or more of the users or teams you specified is not a collaborator of the gardener/etcd-druid repository.".

Additional Information

Redacted in public. Check backend logs.

[seshachalam-yv]

/assign

[ishan16696]

/assign

[shreyas-s-rao]

@seshachalam-yv thanks for your review. Your comments are addressed now. PTAL

[seshachalam-yv]

/lgtm

[ishan16696]

PreSync behaviour to no longer wait for etcd pods to be updated with the latest sts spec, or to be ready

If it's isn't waiting for pod to be ready/running, isn't that can cause a transient quorum loss ?

[ishan16696]

Etcd spec is updated with non-existent secret reference (TLS or backup store)

we usually never go from TLS to non-TLS, Is there any other case where we want to do this ?

[ishan16696]

Had a offline discussion with @shreyas-s-rao and he informed me that he didn't see the transient quorum loss while testing a upgrading from v0.22.3 to v0.23.0, but he observed a transient quorum loss while upgrading v0.22.0 to v0.23.0.
So, we concluded that he will mentioned this in a release note to upgrade the druid to v0.23.0 only atleast from v0.22.3.

[ishan16696]

LGTM!!

[gardener-robot]

@shreyas-s-rao Command "/InviteCommand" failed with "Reviews may only be requested from collaborators. One or more of the users or teams you specified is not a collaborator of the gardener/etcd-druid repository.".

Additional Information

Redacted in public. Check backend logs.

[shreyas-s-rao]

I've added a breaking release note informing users to upgrade to v0.22.3+ before upgrading to v0.23.0.

[gardener-robot]

@shreyas-s-rao Command "/InviteCommand" failed with "Reviews may only be requested from collaborators. One or more of the users or teams you specified is not a collaborator of the gardener/etcd-druid repository.".

Additional Information

Redacted in public. Check backend logs.

[gardener-robot]

@shreyas-s-rao Command "/InviteCommand" failed with "Reviews may only be requested from collaborators. One or more of the users or teams you specified is not a collaborator of the gardener/etcd-druid repository.".

Additional Information

Redacted in public. Check backend logs.