Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added chart hardened Kubelet #6

Merged
merged 21 commits into from
Jun 19, 2024
Merged

Conversation

CerRegulus
Copy link
Contributor

Expose kubelet metrics

Added hardened kubelet to deploy hardened kubelet pushproxy on port 10250.

Motivaiton

Rancher doesn't host kubelet as a pod, so no metrics are directly parsed from it. Since the central DTIT monitoring solution (Dynatrace) needs kubelet metrics to be able to monitor PVCs, we should find a way to expose the kubelet metrics, so that:

  • prometheus can gather them
  • dynatrace has direct access to them as well

Additionally, all prometheuses aren't currently able to parse the kubelet serviceMonitor, even though it's deployed.

Even more additionally, the dynatrace active gate cannot parse the port, as no pod is to be found behind that IP. The logs below are from s01:

org.apache.http.conn.ConnectTimeoutException: Connect to 10.125.40.115:10250 [/10.125.40.115] failed: connect timed out
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)

TODOS

  1. Deploy to staging
  2. Make sure the metrics are exposed on port 10250 of the pushproxy pod
  3. Check that the kubelet serviceMonitor is healthy again in prometheus
  4. Check with the Dynatrace colleagues, that the problem is now solved and the one

@CerRegulus CerRegulus added the enhancement New feature or request label May 23, 2024
@CerRegulus CerRegulus self-assigned this May 23, 2024
@puffitos puffitos self-requested a review May 23, 2024 13:45
@puffitos
Copy link
Member

Please change the version of the chart to 0.10.0-rc1 or something similar, so we can test it in staging. Then you should be able to create a chart release by tagging the main branch.

Cedric Haack and others added 20 commits May 27, 2024 13:31
Signed-off-by: Bruno Bressi <[email protected]>
Signed-off-by: Bruno Bressi <[email protected]>
helm package was creating a chart up to 6MB. This happened because the .git folder was being included in the package.
Signed-off-by: Bruno Bressi <[email protected]>
Signed-off-by: Bruno Bressi <[email protected]>
Signed-off-by: Bruno Bressi <[email protected]>
Signed-off-by: Bruno Bressi <[email protected]>
Signed-off-by: Bruno Bressi <[email protected]>
Signed-off-by: Bruno Bressi <[email protected]>
Signed-off-by: Bruno Bressi <[email protected]>
Copy link
Member

@puffitos puffitos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The kubelet serviceMonitor is getting scraped and everything works as expected. The failing prometheus scrape is because of a certificate problem on some nodes in our development environment.

@CerRegulus CerRegulus merged commit 510d0de into main Jun 19, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants