0.4.0-rc.1 crash

[abh]

I upgraded to 0.4.0-rc.1 because the better "find a master to replicate from" logic sounded like it'd solve problems I've had in the past with the operator. It was running for a little while before crashing with the message below. We were draining and restarting nodes in the cluster while it happened.

{"severity":"INFO","timestamp":"2020-04-24T22:03:54.431386306Z","logger":"controller.orchestrator","message":"update status","key":"ntpdb/ntp-db","diff":["Conditions.slice[0].Message: cluster is writable != writable nodes: ntp-db-mysql-0.mysql.ntpdb","Nodes.slice[2].Conditions.slice[0].Status: True != Unknown","Nodes.slice[2].Conditions.slice[0].LastTransitionTime.Time: 2020-04-24 22:02:59 +0000 UTC != 2020-04-24 22:03:54.430912289 +0000 UTC m=+45.696136914","Nodes.slice[2].Conditions.slice[1].Status: False != Unknown","Nodes.slice[2].Conditions.slice[1].LastTransitionTime.Time: 2020-04-24 22:02:59 +0000 UTC != 2020-04-24 22:03:54.430945824 +0000 UTC m=+45.696170452","Nodes.slice[2].Conditions.slice[2].Status: False != Unknown","Nodes.slice[2].Conditions.slice[2].LastTransitionTime.Time: 2020-04-24 22:02:59 +0000 UTC != 2020-04-24 22:03:54.430972693 +0000 UTC m=+45.696197327"]}
{"severity":"INFO","timestamp":"2020-04-24T22:03:55.634402618Z","logger":"orchestrator-reconciler","message":"cluster not ready for acknowledge","key":"perlweb/perlweb-db","threshold":600}
{"severity":"INFO","timestamp":"2020-04-24T22:03:55.634720895Z","logger":"controller.orchestrator","message":"update status","key":"perlweb/perlweb-db","diff":["Conditions.slice[0].Status: True != False","Conditions.slice[0].LastTransitionTime.Time: 2020-04-24 22:02:56 +0000 UTC != 2020-04-24 22:03:55.63435911 +0000 UTC m=+46.899583735","Conditions.slice[0].Reason: ClusterReadOnlyTrue != ClusterReadOnlyFalse","Conditions.slice[0].Message: cluster is in read only != writable nodes: perlweb-db-mysql-0.mysql.perlweb","Conditions.slice[1].Status: True != False","Conditions.slice[1].LastTransitionTime.Time: 2020-04-24 21:18:23 +0000 UTC != 2020-04-24 22:03:55.634386116 +0000 UTC m=+46.899610754","Conditions.slice[1].Reason: ClusterReady != StatefulSetNotReady","Conditions.slice[1].Message: Cluster is ready != StatefulSet is not ready"]}
E0424 22:03:55.930118       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/github.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:679
/usr/local/go/src/runtime/panic.go:199
/usr/local/go/src/runtime/signal_unix.go:394
/go/src/github.com/presslabs/mysql-operator/pkg/controller/orchestrator/orchestrator_reconcile.go:639
/go/src/github.com/presslabs/mysql-operator/pkg/controller/orchestrator/orchestrator_reconcile.go:149
/go/src/github.com/presslabs/mysql-operator/pkg/controller/orchestrator/orchestrator_reconcile.go:80
/go/src/github.com/presslabs/mysql-operator/vendor/github.com/presslabs/controller-util/syncer/syncer.go:62
/go/src/github.com/presslabs/mysql-operator/pkg/controller/orchestrator/orchestrator_controller.go:209
/go/src/github.com/presslabs/mysql-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
/go/src/github.com/presslabs/mysql-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
/go/src/github.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1357
panic: runtime error: invalid memory address or nil pointer dereference [recovered]