Error initializing cloudprober

[jwillker]

I followed exactly as it is in this tutorial and had this problem to initialize the pod:

1 Error initializing cloudprober. Err: Get "http://169.254.169.254/computeMetadata/v1/project/project-id": dial tcp 169.254.169.254:80: connect: connection refused

In a locally Kubernetes cluster all the things work perfectly but in the GKE clusters not.

I'm using the last version of Cloudprober and Kubernetes(GKE) in v1.14.10-gke.36 version

What I'm doing wrong?

Maybe something related to this:

cloudprober/cmd/cloudprober.go

Lines 129 to 133 in 5c1b573

	// On GCE first check if there is a config in custom metadata
	// attributes.
	if metadata.OnGCE() {
	if config, err := config.ReadFromGCEMetadata(configMetadataKeyName); err != nil {
	glog.Infof("Error reading config from metadata. Err: %v", err)

[jwillker]

As I'm using Istio there are these issues:

• IP no longer allowed in ServiceEntry host field istio/istio#13009
• Stackdriver exporters failing to detect GCE metadata api when run on Istio GoogleCloudPlatform/microservices-demo#199
• Can't connect to Google Cloud Sql istio/istio#6593

The solution for while is:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: metadata-google-internal
spec:
  hosts:
    - metadata.google.internal
  http:
    - route:
        - destination:
            host: metadata.google.internal
          headers:
            request:
             remove:
                - x-forwarded-for
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: metadata-google-internal
spec:
  hosts:
    - metadata.google.internal
  addresses:
    - 169.254.169.254
  ports:
    - name: http
      number: 80
      protocol: HTTP
    - name: https
      number: 443
      protocol: HTTPS
  location: MESH_INTERNAL
  resolution: DNS

But this solution for some reason add a delay of 20s in the pod startup.

[manugarg]

Thanks a lot @JohnWillker for reporting this bug. I haven't seen the istio issues in details yet, but it seems that metadata server is not visible (at least using the IPs which is what Golang libraries use) without adding a virtual service. I'll look into it more to see if we can do something about it.

If you're using the latest cloudprober image, you can try using --cloud_metadata=none flag to disable the cloud metadata initialization (see #417 for more details on this issue).

[jwillker]

Thanks a lot @JohnWillker for reporting this bug. I haven't seen the istio issues in details yet, but it seems that metadata server is not visible (at least using the IPs which is what Golang libraries use) without adding a virtual service. I'll look into it more to see if we can do something about it.

If you're using the latest cloudprober image, you can try using --cloud_metadata=none flag to disable the cloud metadata initialization (see #417 for more details on this issue).

The --cloud_metadata=none flag work for me. Thanks 👍

[manugarg]

I looked around a little bit. Problem is metadata's OnGCE method uses both IP and the metadata hostname both to decide whether it's running on GCE or not:
https://github.com/googleapis/google-cloud-go/blob/59d32d7449b1d48eada528345114be6109965503/compute/metadata/metadata.go#L117

But, while doing the rest of the queries, it uses only IP address:
https://github.com/googleapis/google-cloud-go/blob/59d32d7449b1d48eada528345114be6109965503/compute/metadata/metadata.go#L290

I am not sure what we can do about it in cloudprober. Ideally istio/metadata library should resolve it.

There is a workaround to it though: you can set the environment variable GCE_METADATA_HOST to "metadata.google.internal", or you can simply use the flag --cloud_metadata=none to disable this behavior completely.

[manugarg]

I'll close this. Please let us know if you have any suggestions.