[Advanced StatefulSet] e2e case: delete the failed pod after a failover

[cofyc]

Feature Request

Describe the feature you'd like:

an example:

• (normal) tc.spec.tikv.replicas = 3, sts.spec.replicas = 3
• (failover) tc.spec.tikv.replicas = 3, sts.spec.replicas = 4 ( tc.spec.tikv.replicas + len(tc.status.tikv.failureStores))
• delete the failed pod
  • if the failed pod is not expected to exist, we should delete it from the cc.status.tikv.failureMembers
• (after the deletion) tc.spec.tikv.replicas = 3, sts.spec.replicas = 3 ( tc.spec.tikv.replicas + len(tc.status.tikv.failureMembers))

[cofyc]

I thought about it some more and think this case is meaningless. It's not the right to delete failover replicas. The auto-failover controller adds new replicas by increasing the replicas of the statefulset. According to our best practice, users should delete the entries in tikv.failureStores to decrease desired replicas of tikv statefulset.

[cofyc]

ah, one meaningful use case is to blacklist the failed pod to prevent it from being recreated again when deleting it from tikv.failureStores.