Prometheus metrics \"INVALID\" is not a valid start token in kubernetes env

[jmdinis]

I deployed a keras mnist model using seldon core and I'm trying to monitor the model using seldon-core-analytics with grafana dashaboards. The cluster_monitoring dashboard looks fine but the Prediction Analytics doesn't find the my deployed model and all the panels are empty. I have installed kubeflow, the seldon-core and seldon-core-analytics in the same namespace and I have my model deployed in that same namespace.

After checking prometheus service logs in the same namespace I found the following message recurrently:

level=warn ts=2019-07-17T16:24:27.691294684Z caller=scrape.go:836 component="scrape manager" scrape_pool=kubernetes-pods target=http://192.168.49.198:16686/metrics msg="append failed" err=""INVALID" is not a valid start token"

Can you please advise on this?

Thanks.

[ukclivecox]

I think seldon's scrape path is /prometheus so that looks like its from another pod?

[ukclivecox]

Please reopen if this is still an issue.

[neri4488]

Yes, on my side I added actuator/prometheus.
That was the path, depending on how your service is set up.