getting 'Unable to determine cluster name' in a new cluster

[aberfeldy]

We wanted to move to a different cluster and reusing the same CRD yamls for creating mysql instances via the operator.

apiVersion: mysql.presslabs.org/v1alpha1
kind: MysqlCluster
metadata:
  name: tooling-cluster
  namespace: db
spec:
  replicas: 2
  secretName: tooling-secret
  volumeSpec:
    persistentVolumeClaim:
      resources:
        requests:
          storage: 15Gi

but it just failes with the operator throwing:

{
	"level": "warn",
	"ts": 1577107926.263793,
	"logger": "controller.orchestrator",
	"msg": "can't get instances from Orchestrator",
	"msg": "not found",
	"alias": "tooling-cluster.db",
	"stacktrace": "github.com/presslabs/mysql-operator/vendor/github.com/go-logr/zapr.(*infoLogger).Info\n\t/go/src/github.com/presslabs/mysql-operator/vendor/github.com/go-logr/zapr/zapr.go:69\ngithub.com/presslabs/mysql-operator/pkg/controller/orchestrator.(*orcUpdater).getFromOrchestrator\n\t/go/src/github.com/presslabs/mysql-operator/pkg/controller/orchestrator/orchestrator_reconcile.go:127\ngithub.com/presslabs/mysql-operator/pkg/controller/orchestrator.(*orcUpdater).Sync\n\t/go/src/github.com/presslabs/mysql-operator/pkg/controller/orchestrator/orchestrator_reconcile.go:75\ngithub.com/presslabs/mysql-operator/vendor/github.com/presslabs/controller-util/syncer.Sync\n\t/go/src/github.com/presslabs/mysql-operator/vendor/github.com/presslabs/controller-util/syncer/syncer.go:62\ngithub.com/presslabs/mysql-operator/pkg/controller/orchestrator.(*ReconcileMysqlCluster).Reconcile\n\t/go/src/github.com/presslabs/mysql-operator/pkg/controller/orchestrator/orchestrator_controller.go:205\ngithub.com/presslabs/mysql-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/presslabs/mysql-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/presslabs/mysql-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/presslabs/mysql-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/presslabs/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"
}

the orchestrator tells me:

[martini] Started GET /api/cluster/tooling-cluster.db for 127.0.0.1:37458
2019-12-23 13:41:56 ERROR Unable to determine cluster name. clusterHint=tooling-cluster.db
[martini] Completed 500 Internal Server Error in 18.526038ms
[martini] Started GET /api/audit-recovery/tooling-cluster.db for 127.0.0.1:37458
[martini] Completed 200 OK in 4.251511ms
[martini] Started GET /api/raft-health for 116.202.10.69:52714
[martini] Completed 200 OK in 1.092799ms

The first pod is created and most of the containers are ready, but the mysql container is marked as not ready. Thus the cluster won't be created and is not usable.
I've installed the operator via helm with the version v0.3.1

[aberfeldy]

Seems to be an RBAC problem, if the operator is deployed in different namespace (e.g. default) than the cluster (eg. database). Both in the same namespace work fine

[AMecea]

The cluster can't be registered into orchestrator, that seems to be the problem. Make sure that the orchestrator (operator) can access the pods. Do you have some custom configuration for k8s?

[aberfeldy]

Never mind. figured out that our Rancher had some network policies enabled which refused connections from other project.