Panic when creating mesh and virtual nodes at the same time

[stefanprodan]

When creating a mesh and virtual nodes, it takes some time for the mesh to become active and the operator panics. I would label this as a bug since the mesh controller should wait for the mesh to become active and not crash the whole process.

I0313 12:52:26.429780   34169 controller.go:208] Mesh Added
I0313 12:52:26.633567   34169 controller.go:244] Mesh Updated
I0313 12:52:30.465397   34169 controller.go:260] Virtual Node Added
E0313 12:52:30.668255   34169 controller.go:370] error syncing 'test/backend': mesh global must be active for virtual node backend
I0313 12:52:30.668744   34169 controller.go:268] Virtual Node Updated
E0313 12:52:30.670646   34169 controller.go:370] error syncing 'test/backend': mesh global must be active for virtual node backend
I0313 12:52:31.200551   34169 controller.go:260] Virtual Node Added
E0313 12:52:31.401782   34169 controller.go:370] error syncing 'test/backend-primary': mesh global must be active for virtual node backend-primary
I0313 12:52:31.401984   34169 controller.go:268] Virtual Node Updated
E0313 12:52:31.402028   34169 controller.go:370] error syncing 'test/backend-primary': mesh global must be active for virtual node backend-primary
E0313 12:52:31.636186   34169 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/Users/aleph/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:76
/Users/aleph/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:65
/Users/aleph/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:51
/usr/local/Cellar/go/1.11.4/libexec/src/runtime/asm_amd64.s:522
/usr/local/Cellar/go/1.11.4/libexec/src/runtime/panic.go:513
/usr/local/Cellar/go/1.11.4/libexec/src/runtime/panic.go:82
/usr/local/Cellar/go/1.11.4/libexec/src/runtime/signal_unix.go:390
/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/aws/appmesh.go:69
/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/mesh.go:44
/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:308
/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:359
/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:367
/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:325
/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:308
/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:164
/Users/aleph/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
/Users/aleph/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:134
/Users/aleph/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
/usr/local/Cellar/go/1.11.4/libexec/src/runtime/asm_amd64.s:1333
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x1788c1e]

goroutine 59 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/Users/aleph/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:58 +0x108
panic(0x1f6b340, 0x2e1a3d0)
	/usr/local/Cellar/go/1.11.4/libexec/src/runtime/panic.go:513 +0x1b9
github.com/aws/aws-app-mesh-controller-for-k8s/pkg/aws.(*Cloud).GetMesh(0xc000322630, 0x2316180, 0xc0001fe360, 0xc0004a2ff0, 0x6, 0x0, 0x0, 0x0)
	/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/aws/appmesh.go:69 +0x18e
github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller.(*Controller).handleMesh(0xc0003409a0, 0xc0004a3070, 0xb, 0x18, 0xc0004c6d18)
	/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/mesh.go:44 +0x226
github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller.(*Controller).handleMesh-fm(0xc0004a3070, 0xb, 0xc00041ad80, 0x1ef49e0)
	/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:308 +0x3e
github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller.processNextWorkItem.func1(0x23263e0, 0xc00041ad80, 0xc0004c6e78, 0x1ef49e0, 0xc00066ade0, 0x0, 0x0)
	/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:359 +0xd5
github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller.processNextWorkItem(0x23263e0, 0xc00041ad80, 0xc0004c6e78, 0x0)
	/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:367 +0x75
github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller.(*Controller).processNext(0xc0003409a0, 0x23263e0, 0xc00041ad80, 0xc0004c6e78, 0xc0004c6ea0)
	/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:325 +0x3f
github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller.(*Controller).meshWorker(0xc0003409a0)
	/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:308 +0x68
github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller.(*Controller).meshWorker-fm()
	/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:164 +0x2a
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0005f8df0)
	/Users/aleph/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x54
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0005f8df0, 0x3b9aca00, 0x0, 0x1, 0x0)
	/Users/aleph/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:134 +0xbe
k8s.io/apimachinery/pkg/util/wait.Until(0xc0005f8df0, 0x3b9aca00, 0x0)
	/Users/aleph/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller.(*Controller).Run
	/Users/aleph/go/src/github.com/aws/aws-app-mesh-controller-for-k8s/pkg/controller/controller.go:164 +0x2ff

[stefanprodan]

When creating a mesh without any virtual nodes the operator runs into a race condition:

I0313 13:24:25.012794   34468 controller.go:208] Mesh Added
I0313 13:24:25.226264   34468 controller.go:244] Mesh Updated
I0313 13:24:26.431067   34468 mesh.go:55] Created mesh global
E0313 13:24:26.643118   34468 controller.go:370] error syncing 'test/global': error updating mesh status: Operation cannot be fulfilled on meshes.appmesh.k8s.aws "global": the object has been modified; please apply your changes to the latest version and try again
I0313 13:24:26.874109   34468 mesh.go:57] Discovered mesh global
I0313 13:24:27.181350   34468 controller.go:365] Successfully synced 'test/global'
I0313 13:24:27.181412   34468 controller.go:244] Mesh Updated
I0313 13:24:27.408034   34468 mesh.go:57] Discovered mesh global
I0313 13:24:27.408064   34468 controller.go:365] Successfully synced 'test/global'
I0313 13:24:42.861723   34468 controller.go:244] Mesh Updated
I0313 13:24:43.154828   34468 mesh.go:57] Discovered mesh global
I0313 13:24:43.154874   34468 controller.go:365] Successfully synced 'test/global'