GCE Ingress fails to sync instance group in multi-az cluster

[petercm]

When an ingress is spun up in a multi-zone cluster, no nodes end up in the k8s-ig instance group. My logs show

I0513 12:21:19.826319 1 instances.go:130] Node pool encountered a 404, ignoring: googleapi: Error 404: The resource 'projects/myproject/zones/us-east1-c/instances/gke-mazcluster-default-pool-67470956-1n15' was not found, notFound

(gke-mazcluster-default-pool-67470956-1n15 is not found in us-east1-c because it is in us-east1-d)

I believe the underlying issue here relates to getting the gce cloudprovider without config:

In ClusterManager.getGCEClient()

The GCE provider is retrieved with nil as the config param.
cloudInterface, err := cloudprovider.GetCloudProvider("gce", nil)

Which means cfg.Global.Multizone would never get set, managedZones will not get set to nil and instead we will always be looking in the primary zone.

I believe the fix should be similar to kubernetes/kubernetes#23769 but I don't know how you might/if it's possible to get a reference to the kubelet and configured CloudProvider here vs. how it can be done in a plugin.

@bprashanth

[bprashanth]

@petercm thanks for the report, yeah I debugged it recently and came to a similar conclusion, just haven't had a chance to fix it.

When an ingress is spun up in a multi-zone cluster

You mean "ubernetes-lite", yes? just making sure I understand the setup.

Honestly both ubernetes-lite and ingress were developed in beta, in parallel, so I'm not surprised they don't play well together. We should definitely make this work.

If you have some cycled, please send a pr. I think there's at least one more issue that needs to be dealt with, and that's to pool instances in per-zone instance groups, and add such instance groups to the list of backends in a backend-service. Today the controller assumes all instances are in that one zone and creates a single IG for them.

[petercm]

Yes, "ubernetes-lite". I haven tried manually doing the multi-zone config as shown on the blog

I'm assuming thats what gcloud container clusters create mycluster --zone zone1 --additional-zones zone2,zone3 is doing under the covers.

I suspected per-zone grouping would also be a thing but hadn't really looked at it. I suppose since people may choose a subset of zones the work to set up groups will have to be done -- otherwise I'd opt to look at regional instance groups

I'm not sure if I'll be able to get to it before you might but I'll try and find time to give it a shot. As I mentioned though I don't know where to start with getting a ref to the configured CloudProvider on the kubelet.

[bprashanth]

I have a fix, hopefully I get time to test it enough to include in 1.3 but no promises https://github.com/kubernetes/contrib/pull/1032/files. Even if it doesn't make 1.3 we can push it out in a minor release, though.