Addon: Use Hostprocess for Windows Exporter #2048
Conversation
dineshsharmads
changed the title
|
@dineshsharmads - sorry for the lag on this, do you mind rebasing and I can trigger CI? |
This allows for Windows Exporter to be deployed and configured dynamically without requiring to specify targets manually for each Windows node Documentation and example added
Fix some performance issues. - Specify resource limits for windows exporter. - Allow for scrape timeout and interval to be configured. Depending on how many pods are running on a node it can take from 500ms to 15s to scrape metrics from node. Default timeout is 10s. - Allow for enabled collectors to be configured. - Only enable collectors that are being used in rules and dashboards.
dineshsharmads
force-pushed
the
hostprocess-windows-exporter
branch
from
e41a3c4
to
ae1c2dd
Compare
|
@philipgough no worries, I have rebased. |
docs/windows.md
Outdated
| @@ -21,3 +21,20 @@ local kp = (import 'kube-prometheus/main.libsonnet') + | |||
| ``` | |||
|
|
|||
| [Containerd](https://github.com/prometheus-community/windows_exporter/blob/master/kubernetes/kubernetes.md) version can run as pod. | |||
|
|
|||
| ``` | |||
| local kp = (import 'kube-prometheus/main.libsonnet') + | |||
There was a problem hiding this comment.
Can we update the docs above that say Currently, Docker based Windows does not support running with [windows_exporter](https://github.com/prometheus-community/windows_exporter) in a pod so this add on uses [additional scrape configuration](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md) to set up a static config to scrape the node ports where windows_exporter is configured.?
I would also lean towards moving this up in the docs and making it the default over the static pod configuration. We should drop the static pod configuration in the future.
There was a problem hiding this comment.
Sure, I have updated the doc to reflect both configs and moved hostprocess config up.
| image:: error 'must provide version', | ||
| resources:: { | ||
| requests: { cpu: '300m', memory: '200Mi' }, | ||
| limits: { memory: '200Mi' }, |
There was a problem hiding this comment.
how did you choose these vaules? They might be a bit low for Windows?
There was a problem hiding this comment.
I'm running it from my fork in few windows clusters.
I see the usage as below. The scrape duration that I'm getting for windows exporters is between 400-700ms.
NAME CPU(cores) MEMORY(bytes)
windows-exporter-2lw7j 24m 31Mi
windows-exporter-8vxc9 22m 31Mi
windows-exporter-bbpc8 17m 31Mi
windows-exporter-c7qgw 22m 31Mi
windows-exporter-gmq4p 15m 29Mi
windows-exporter-r4zbk 21m 30Mi
windows-exporter-r7kms 17m 29Mi
windows-exporter-rhrfn 19m 30Mi
windows-exporter-t2r9l 21m 29Mi
windows-exporter-tb699 19m 32Mi
windows-exporter-tmbz2 19m 30Mi
windows-exporter-xk8n6 21m 32Mi
There was a problem hiding this comment.
Is that idle or with pods running on the system? Just want to make sure we aren't too low out of the box, I've seen other windows "system" pods hit OOM limits when under load. Based on what you shared, this looks reasonable to start with.
There was a problem hiding this comment.
This is a non-prod cluster but it has 60 pods per e8s node. I just deployed it in our prod cluster which is running 35 pods per node but has higher CPU utilization. And I see similar numbers. I'm guessing since it is a host process pod it does not spin up other system services per pod (csrss,svchost etc). So the utilization is just for windows_exporter process.
There was a problem hiding this comment.
I'm guessing since it is a host process pod it does not spin up other system services per pod (csrss,svchost etc). So the utilization is just for windows_exporter process.
This makes sense. Thanks for verifying!
- Hostprocess and static
|
/assign @philipgough |
|
Thanks for the review @jsturtevant @dineshsharmads Im no windows expert, but the change looks fine. I'll need to fix the broken CI and we can merge. |
|
Actually, will merge this first to avoid you having to circle back on it. Thanks |
|
Thanks @philipgough and @jsturtevant. |
Description
This PR adds an addon
windows-hostprocess.libsonnetwhich deploys and configures Windows Exporter as a hostprocess pod on all Windows nodes dynamically.The existing addon
windows.libsonnetrequires a static list of node IPs to be provided.This deploys a daemonset and podmonitor . It basically implements the setup mentioned in windows_exporter on Kubernetes with some minor fixes. (enable memory collector, relabel node name, use windowsExporterSelector instead of wmiExporterSelector).
I have a few AKS clusters with Windows nodes where I have tested the addon and dashboards.
Addresses #1627
Type of change
CHANGE(fix or feature that would cause existing functionality to not work as expected)FEATURE(non-breaking change which adds functionality)BUGFIX(non-breaking change which fixes an issue)ENHANCEMENT(non-breaking change which improves existing functionality)NONE(if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)Changelog entry