unable to update list of all metrics: unable to fetch metrics for query when using default configuration

[aaronmell]

I'm using the latest helm chart. I compared the configuration values with the one's in this project and they match. Running the latest image.

I0816 16:24:37.281904 1 serving.go:273] Generated self-signed cert (/tmp/cert/apiserver.crt, /tmp/cert/apiserver.key)
I0816 16:24:37.974351 1 serve.go:96] Serving securely on [::]:6443
E0816 16:24:38.211221 1 provider.go:281] unable to update list of all metrics: unable to fetch metrics for query "{name="^container_.*",container_name!="POD",namespace!="",pod_name!=""}": bad_response: invalid character '<' looking for beginning of value
E0816 16:25:08.594633 1 provider.go:281] unable to update list of all metrics: unable to fetch metrics for query "{namespace!="",name!"^container_.*"}": bad_response: invalid character '<' looking for beginning of value
I0816 16:25:10.744735 1 wrap.go:42] GET /healthz: (14.7782ms) 200 [[kube-probe/1.10] 10.1.0.1:53258]
I0816 16:25:11.060037 1 wrap.go:42] GET /healthz: (1.1777ms) 200 [[kube-probe/1.10] 10.1.0.1:53268]
I0816 16:25:11.894400 1 authorization.go:73] Forbidden: "/", Reason: ""
I0816 16:25:11.894830 1 wrap.go:42] GET /: (1.9933ms) 403 [[Go-http-client/2.0] 192.168.65.3:39348]
I0816 16:25:11.899790 1 authorization.go:73] Forbidden: "/", Reason: ""
I0816 16:25:11.899936 1 wrap.go:42] GET /: (259.7µs) 403 [[Go-http-client/2.0] 192.168.65.3:39348]
I0816 16:25:12.576370 1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (740.5µs) 200 [[kubectl/v1.10.3 (darwin/amd64) kubernetes/2bba012] 192.168.65.3:39388]
I0816 16:25:14.409052 1 wrap.go:42] GET /openapi/v2: (2.2461ms) 404 [[] 192.168.65.3:39388]
I0816 16:25:14.412749 1 wrap.go:42] GET /swagger.json: (2.4773ms) 404 [[] 192.168.65.3:39388]

[aaronmell]

Looks like this is an issue on my side still. I am still not hitting the correct endpoint for prometheus

[arvin6msd]

I seem to have the same problem. But the error is different.
E0714 13:50:24.799479 1 provider.go:209] unable to update list of all metrics: unable to fetch metrics for query "{__name__=~\"^container_.*\",container!=\"POD\",namespace!=\"\",pod!=\"\"}": bad_response: unknown response code 404
I'm pointing to the right url as well
--prometheus-url=http://msd-promop-v1-prometheus-o-prometheus.monitoring.svc:9090

Pretty much stuck with getting the default configuration to work. Can someone help?

[mathe-matician]

@aaronmell

Looks like this is an issue on my side still. I am still not hitting the correct endpoint for prometheus

Can you clarify what you mean by this? My issue is that my custom metrics work, and then when they scale to a certain point, the HPA goes into an unknown state.

This is pretty interesting: stripe/stripe-go#642 (comment):

I suspect that < is the start of