-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cAdvisor metrics stopped working correctly in K3s 1.20 #2831
Comments
Have you tried v1.20.2? Possibly related to kubernetes/kubernetes#97006 Although the fact that overriding the arg resolves it is interesting. |
Upgrading to v1.20.2+k3s1 (1d4adb0) didn't help, still need that extra kubelet arg for proper labels. The only change I noticed was presence of additional metrics, presumably added by kubernetes/kubernetes#97006 |
Great! It is working for me also by adding kubelet-arg. 👍 |
same here. Thanks a lot for the workaround. |
Issue was reproducible on k3s v1.20.2+k3s1
Validated metrics have non-empty labels using commit id k3s version v1.20.2+k3s-c5e2676d
|
Sorry for commenting on a closed issue, but I am using
|
@lackhoa the fix isn't in v1.20.2+k3s1. QA tested on a post-release CI build (v1.20.2+k3s-c5e2676d) off master. That commit is included in v1.20.4+k3s1; please use that. Not all kubelet args are documented; for historical reasons all cadvisor args are also valid kubelet args, despite their not being in the docs. |
@brandond Ah thanks, I'll try that. |
I think the |
Yeah, it's one of those things that make it clear that dockershim is still the only thing that upstream actually tests, despite all the big talk about deprecating it. You run into all kinds of weird issues if you actually use a different runtime. |
Alright, I can confirm that my problem has been resolved after upgrading to |
You don't need to add the containerd arg on v1.20.4. |
Are there docs for how to enable these metrics? I'm looking to use a dashboard that makes use of a metric |
I'm running v1.24.2+k3s2 and still have to pass |
@discordianfish No, you're just on an old version of K3s. Update to the latest 1.24 patch release. |
Environmental Info:
K3s Version:
k3s version v1.20.0+k3s2 (2ea6b16)
go version go1.15.5
Node(s) CPU architecture, OS, and Version:
Linux kube-master0 5.4.0-54-generic #60-Ubuntu SMP Fri Nov 6 10:37:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration:
1 master, 2 workers, embedded containerd (no docker, no custom CRI)
Describe the bug:
cAdvisor is unable to connect to containerd resulting in mostly empty labels (container, image, name) in metrics.
Adding
--kubelet-arg containerd=/run/k3s/containerd/containerd.sock
to k3s launch args fixes the issue.Steps To Reproduce:
Expected behavior:
Metrics from cAdvisor have non-empty labels:
Actual behavior:
cAdvisor metric's labels are empty
Additional context / logs:
Most likely regression caused by 5b318d0
That value is used for both
argsMap["container-runtime-endpoint"]
andargsMap["containerd"]
and it seems the containerd one cannot be an URIThe text was updated successfully, but these errors were encountered: