[stable/newrelic-infrastructure] Unable to discover kube-state-metrics due to label change

[adamcroissant]

Describe the bug
The newrelic infrastructure agent will not result in the cluster appearing in newrelic currently if installed clean because it cannot auto-locate the kube-state-metrics service and pods. Exact error message is:

level=panic msg=\"failed to discover nodeIP with kube-state-metrics, got error: no pods found by any of labels k8s-app, app with value kube-state-metrics\"\ntime=\"2019-07-25T16:03:02Z\" level=fatal msg=\"failed to discover nodeIP with kube-state-metrics, got error: no pods found by any of labels k8s-app, app with value kube-state-metrics\"\n"

Version of Helm and Kubernetes:
Helm: v2.11.0
Kubernetes: v1.11.8

Which chart:
stable/newrelic-infrastructure

What happened:
Helm install succeeded but data never appeared in NewRelic

What you expected to happen:
Data to appear in NewRelic within five minutes

How to reproduce it (as minimally and precisely as possible):
Install the latest version of stable/kube-state-metrics into a cluster, and then install the latest version of stable/newrelic-infrastructure.

Go check in your newrelic account and observe that the cluster is not listed within Infrastructure -> Kubernetes.

Check the logs of one of the newrelic-infrastructure pods and note the error pasted above.

Anything else we need to know:
A workaround is to patch the labels on the kube-state-metrics service and deployment to match what newrelic-infrastructure is looking for, though this is untenable long term due to it requiring effort after every update of kube-state-metrics.

kubectl patch svc kube-state-metrics --type='json' \
  --patch="[{'op':'add','path':'/metadata/labels/app','value':'kube-state-metrics'}]" 

kubectl patch deployments kube-state-metrics --type='json' \
  --patch="[{'op':'add','path':'/spec/template/metadata/labels/app','value':'kube-state-metrics'}]" 

The breaking change was introduced in kube-state-metrics here: #15261

[douglascamata]

Hi @adamcroissant, thanks for the report.

I work on the New Relic k8s integration and we will get a new release out in the next couple of days that is capable of discovering KSM with this new label.

Meanwhile, your proposed workaround works. Another workaround could be also using the KUBE_STATE_METRICS_URL environment variable in the k8s integration container, as described in our docs.

I suggest that we close this issue as it's New Relic that needs to fix their k8s integration and not Helm that needs to fix the chart, because the new label is the "standard" way of describing app names.

[douglascamata]

@adamcroissant release v1.9.3 is done with support to the new label added by the Helm chart: https://docs.newrelic.com/docs/release-notes/platform-release-notes/host-integrations-release-notes/new-relic-integration-kubernetes-193