Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hard coded metadata endpoint breaks logging under GKE managed Istio #7517

Closed
rsalmond opened this issue Mar 15, 2019 · 0 comments
Closed

Hard coded metadata endpoint breaks logging under GKE managed Istio #7517

rsalmond opened this issue Mar 15, 2019 · 0 comments
Assignees
Labels
api: logging Issues related to the Cloud Logging API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@rsalmond
Copy link
Contributor

rsalmond commented Mar 15, 2019

Environment details

  1. API: GCE / GKE internal metadata endpoint
  2. OS: anything running in GKE with managed istio enabled.
  3. Python version: 3.7.2 (container, no venv)
  4. google-cloud-logging version: 1.9.0 (out of date but I double checked on master branch and metadata hostname is present there too).

Steps to reproduce

(please bear with me, I do get to the point!)

  1. Launch a GKE cluster and tick the managed istio box to enable it.
  2. Add the label istio-inejction: enabled to the default namespace to allow istio to control traffic on workloads deployed there.
  3. Launch a container into the namespace which will now have the istio/envoy sidecar automatically added to it. (eg. kubectl run -i --tty busybox --image=busybox --restart=Never -- sh)
  4. Attempt to access the hostname metadata.
wget http://metadata/computeMetadata/v1/instance/attributes/cluster-name --header "Metadata-Flavor: Google"
Connecting to metadata (169.254.169.254:80)
wget: server returned error: HTTP/1.1 404 Not Found

This is the result of Istio's default policy of blocking all outbound requests until they are whitelisted.

Since istio/istio#10694 will not accept non FQDN's the hardcoded metadata endpoint in the python logging client fails.

This manifests as logs showing up in stackdriver without the expected GCE / GKE annotations of instance name, cluster name, etc.

The approach from other google libraries (eg. googleapis/google-cloud-java#4269 ) is to update to the fully qualified metadata.google.internal or just hit the magic IP address directly which Istio has no problems whitelisting.

@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label Mar 16, 2019
@tseaver tseaver added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: logging Issues related to the Cloud Logging API. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Mar 18, 2019
@yoshi-automation yoshi-automation removed the triage me I really want to be triaged. label Mar 18, 2019
tseaver pushed a commit that referenced this issue Mar 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: logging Issues related to the Cloud Logging API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

3 participants