cAdvisor-less, CRI-full Container and Pod Stats

[haircommander]

Enhancement Description

• One-line enhancement description (can be used as a release note): cAdvisor-less, CRI-full Container and Pod Stats
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2371-cri-pod-container-stats/README.md
• Primary contact (assignee): @haircommander, @bobbypage
• Responsible SIGs: sig-node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.26
  • Beta release target (x.y): 1.28
  • Alpha release target (x.y): 1.29 See KEP 2371: retarget to alpha for 1.29 #4249
  • Beta release target (x.y): 1.32
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    • KEP-2371: update kep to reflect current state of enhancement #2812
    • KEP-2371: Add cgroup metrics + CRI implementation plan #3559
    • Add Windows pod sandbox stats information to KEP 2371 - cri pod container stats #3439
    • CRI: Add Windows Podsandbox Stats kubernetes#110754
    • kubelet: add support for broadcasting metrics from CRI kubernetes#113609
  • Code (k/k) update PR(s):
    • 1.23 Kubelet: implement support for podAndContainerStatsFromCRI kubernetes#103095
    • 1.26 CRI: Add Windows Podsandbox Stats kubernetes#110754
  • Docs (k/website) update PR(s): Document PodAndContainerStatsFromCRI feature website#30535
• Beta 1.28
  • KEP (k/enhancements) update PR(s): KEP 2371: target to beta for 1.28 #3770
  • Code (k/k) update PR(s):
    • e2e_node: add container metrics from cadvisor test kubernetes#118838
    • 1.28 kubelet: Implement support for Windows podAndContainerStatsFromCRI kubernetes#116968
  • Docs (k/website) update PR(s):
• Alpha 1.29
  • KEP (k/enhancements) update PR(s): KEP 2371: retarget to alpha for 1.29 #4249
  • Code (k/k) update PR(s):
    • 1.30 [KEP-2371] Fixing bug with windows criStatsProvider sometimes not reporting AvailableBytes correctly kubernetes#122846
  • Docs (k/website) update PR(s):
• Beta 1.32
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
    • Kubelet: Fix ephemeral storage limits not consider termination log kubernetes#126006
  • Docs (k/website) update PR(s):

[haircommander]

/sig node

[ehashman]

/milestone v1.21

[JamesLaverack]

Hey @haircommander and @bobbypage, enhancements 1.21 shadow here,

Enhancements Freeze is 2 days away, Feb 9th EOD PST

The enhancements team is aware that KEP update is currently in progress (PR #2364). Please make sure to work on PRR questionnaires and requirements and get it merged before the freeze. For PRR related questions or to boost the PR for PRR review, please reach out in Slack on the #prod-readiness channel.

Any enhancements that do not complete the following requirements by the freeze will require an exception.

• [IN PROGRESS] The KEP must be merged in an implementable state
• [IN PROGRESS] The KEP must have test plans
• [IN PROGRESS] The KEP must have graduation criteria
• [IN PROGRESS] The KEP must have a production readiness review

[annajung]

Hi @haircommander and @bobbypage, 1.21 Enhancements Lead here.

Enhancements Freeze is now in effect.

Unfortunately, your KEP needed to be updated and the PR has not yet merged. If you wish to be included in the 1.21 Release, please submit an Exception Request as soon as possible.

/milestone clear

[ehashman]

/milestone v1.22

[salaxander]

Hey @haircommander and @bobbypage - 1.22 enhancements team here! Just a reminder that enhancements freeze is coming up on 5/13. We'll need the KEP merged before then to get this included in the 1.22 release.

Let us know if there's anything we can do to help before then!

[salaxander]

@haircommander @bobbypage #2364 merged so we've got you tracked for 1.22 :)

[PI-Victor]

Hello @haircommander @bobbypage 👋, 1.22 Docs release lead here.
This enhancement is marked as ‘Needs Docs’ for 1.22 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.22 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Fri July 9, 11:59 PM PDT.
 Also, take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thank you!

[haircommander]

thanks for the heads up @PI-Victor !

[ehashman]

xref kubernetes/kubernetes#102789

[haircommander]

xref initial kubelet implementation kubernetes/kubernetes#103095

[salaxander]

Hey @bobbypage and @haircommander - Just checking in as we're about 2 weeks away from 1.22 code freeze. I've got kubernetes/kubernetes#103095 and kubernetes/kubernetes#103095 tracked as the open k/k PRs. Are there any other open or merged PRs we should be tracking? Thanks!

[haircommander]

none opened yet! I will post them here if we do. Thanks for your work @salaxander

[salaxander]

Hey @haircommander - One more check-in as we're a week out from 1.22 code freeze. Any updates on if you expect kubernetes/kubernetes#103095 and kubernetes/kubernetes#103095 to merge before the deadline?

Thanks!

[haircommander]

Thanks for checking! I do expect them to merge. We're waiting on a e2e POC which I'm working on :)

[salaxander]

Hi @haircommander - One last ping (sorry!). Code freeze is tomorrow evening (PST), so those two open PRs will need to merge before then for this to be included in 1.22. Let me know if there's anything we can do to help :)

[haircommander]

As a note: I think we are deferring this (again 🙃 ) from 1.30 until containerd and cri-o have support. The cri-o support is actively being worked on, and containerd is being taken by @adisky . I am hoping to make forward progress in 1.31

[ialidzhikov]

yeah containerd needs to be extended to pass the required container and pod metrics through to the kubelet. That said, for metrics that weren't formerly exposed by cadvisor (like the pull metrics that this comment was done on), it may be more efficient to continue to read them directly from containerd

Hi @haircommander,

Just to make sure I understand correct: This KEP won't help with exposing containerd metrics (like the image pulls ones as above) via kubelet, right? Hence, if we need the containerd metrics for the image pulls, we need somehow to collect/scrape these metrics from containerd directly, right?

[haircommander]

yes correct!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[bitoku]

/assign

[pacoxu]

@haircommander
I update the issue description according to kep and PR. Please check if I misunderstand about the current status of this feature.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[HirazawaUi]

/remove-lifecycle rotten

[kannon92]

@haircommander Can you give an update on your plans for 1.32? Do you want this to included in KEP feature planning for 1.32?

[haircommander]

I'd love to make progress in 1.32. it in part relies on containerd support but maybe we can decouple those. let's plan on it for 1.32

[HirazawaUi]

@haircommander I'm curious whether we need to ensure consistent behavior when enabling or disabling the feature gate. If it's not necessary, then we can push kubernetes/kubernetes#126006 forward.

[akhilerm]

I'd love to make progress in 1.32. it in part relies on containerd support but maybe we can decouple those. let's plan on it for 1.32

@haircommander Is ListPodSandboxMetrics the only pending piece from containerd. I should be able to get that in by 1.32

[haircommander]

my hero! let's try to push this to beta in 1.32 then

[pacoxu]

Xref WIP KEP #4792 which may be related.

[zvonkok]

@haircommander Is this still on your agenda? What is the plan here?

[haircommander]

yup mostly waiting on containerd/containerd#10691 which @akhilerm is working on. Once containerd has an implementation we will push this to beta

[zvonkok]

@haircommander The original KEP says under Goals: "Enhance CRI implementations to provide metrics analogous to the existing metrics provided by /metrics/cadvisor.". Comparing https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md and https://github.com/containerd/containerd/pull/10691/files shows me we're missing some metrics in the PR. Do we have a reference list of which metrics need to be available to move this forward?