With PodUID after Pod dies, several Pods in isoGroups become unavailable

[rongo89]

I have an m3db cluster with 3 statefulsets(3 isolationGroups) and 3 nodes each. The podIdentifySource is PodUID because local disk way faster in our case.

• What version of the operator are you running? Please include the docker tag. If using master, please include the git

  • v0.3.0
  • docker-pullable://quay.io/m3db/m3db-operator@sha256:bdd6e3b89eb634f7452119504411e4009088abedce707231b524116ed85152e7

• What version of Kubernetes are you running? Please include the output of kubectl version.

  • v1.14.8

• What are you trying to do?

  • one Pod of an statefulset is deleted on purpose.

• What did you expect to happen?

  • New Pod should came up and stream from rest of the of the isolation group members.

• What happened?
  New Pod came up (in statefulset where pod was deleted before). At the same moment another pod of the group becomes unavailable and starts to stream from somewhere else. Also in the other isolationGroups one Pod each becomes unavailable and starts to stream from other members.