Remove/propose different "no metrics known for pod" log

[serathius]

There are a lot of issues where users are seeing metrics-server reporting error "no metrics known for pod" and asking for help.

To my understanding this error is expected to occur in normal healthy metrics-server.
Metrics Server periodically scrapes all nodes to gather metrics and populate it's internal cache. When there is a request to Metrics API, metrics-server reaches to this cache and looks for existing value for pod. If there is no value for existing pod in k8s, metrics server reports error "no metrics known for pod". This means this error can happen in situation when:

• fresh metrics-server is deployed with clean cache
• query is about fresh pod/node that was not yet scraped

Providing better information to users would greatly reduce throughput of tickets.

[serathius]

/help

[k8s-ci-robot]

@serathius:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tomknock]

There is a same metric-server problem in my cluster: unable to fetch pod metrics for pod kube-system/POD_NAME:no metrics known for pod.
Also,Dashboard couldn't be deployed successfully on k8s:v1.16.2 without metric/heapster: No metric client provided. Skipping metrics.

[zhangyu84848245]

@serathius

I had the same problem before。
add

resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 5m
memory: 50Mi
I can run it， You try。

[serathius]

Hey @zhangyu84848245,
This message is related metrics-server not having cache pre-filled. Increasing resources like you suggested can reduce chances of log message appearing, but will not fully remove it (additional cpu will shorten time needed to generate self-signed cert)

[serathius]

/assign

[serathius]

Possible solutions (in order of implementation complexity):

1. [Preferred] Don't treat missing metrics as errors (remove log calls)
2. Dont' report error for newly created nodes/containers (time.Now().Sub(startTime) < metricResolution + cAdvisorHousekeepingTime)
3. Don't report error if node with metrics was not scraped before container start (keep start time per node and check scrapeTime.Sub(startTime) < cAdvisorHousekeepingTime)

Where cAdvisorHousekeepingTime = 15s

I propose to remove error logs, as those error logs can be caused by:

• Failure in scraping. Still they don't provide useful for debugging, still requiring to read scrape failure logs.
• Metric availability delay. Misinforming that there is a problem with pipeline, instead informing about this being expected behavior.

Other ways we can improve visibility in metric availability delay:

• Document what is expected delay of metrics from freshly created containers and nodes.
• Create histogram metric that measures freshness of served metrics.

Both option 2 & 3 try to add logic to guess health of metrics pipeline. They complicate code without providing any additional benefits. Measuring health of pipeline should be done via defining proper metrics and defining externally monitored SLOs.

/cc @s-urbaniak
Do you agree with this approach?

[serathius]

/cc @kawych

[JoseThen]

Thank you for this @serathius , I was concerned about the issue but noticed it would stop logging after some time, everything is so far looking good 🙇

[serathius]

ping @s-urbaniak

[s-urbaniak]

I agree with just going forward with option 1. I think options 2. and 3. should be solved via a higher level alerting system.

[serathius]

Looks like work was done