Collection timed out, still waiting for

[Monolithys]

Hello! I see some same issues , but can't understand why it is happens, and most of all how decided it.
I using it on OS Windows 2012 R2 Standart, with not so bad hardware Xeon E31230 with 8gb RAM
Running WE as service
"C:\Program Files\windows_exporter\windows_exporter.exe" --collectors.enabled cpu,cs,iis,logical_disk,logon,memory,os,net,service,system,tcp --collector.service.services-where "Name='windows_exporter'"

Before try as APP and have same problems:
every 16 minutes some of collector do not answer for 3 min. And it is happens all times, as it was be scheduled

I was received this errors:

time="2020-12-16T22:22:42+03:00" level=warning msg="Collection timed out, still waiting for [cs os service iis]" source="exporter.go:202"
time="2020-12-16T22:22:50+03:00" level=warning msg="Collection timed out, still waiting for [logon iis os service cs]" source="exporter.go:202"

Other collector still working. But in this moment (for 3 min) "WMI Provider Host" consumed 13-14% of CPU
Logs does not show anything (i mean no errors)
I understand it may be my problem, but hope you can help me with this problem

[Monolithys]

As i understand i must set something like this on Prometheus server

  - job_name: 'windows_exporter'
    scrape_interval:   30s
    static_configs:
    - targets: ['s2:9182']

Srange, standard scrape_interval: sets to 15s

Or i must used 60 seconds?
ADD: Try 60 seconds - same problem. 16 min work, 3 min timed out.

[E1v1s]

The same issue on WS 2016
cpu load is low
trying to increase --scrape.timeout-margin but without results
and try to change policy of startup of wii performance adapter

[Monolithys]

I try it too, nothing changes. Try to get help from team.

[E1v1s]

We are solve problem by increasing scrape time in prometheus from 2s to 5
It's unclear for me but its work stable for 2 days now.

[Monolithys]

@E1v1s
Hi!
May be I did not understand clear, but I try to set
scrape_interval: 60s
But this did not helped any

If it is not difficult - show part from you config at Prometheus

[E1v1s]

  endpoints:
  - interval: 10s
    port: metrics
    scrapeTimeout: 5s

it's a part of config from prometheus in k8s

[Monolithys]

For me it does not helped. Some work normally, and some came with stable delay

[caugner]

Same issue here, on a server with Windows Server 2016, 64 GB memory and 16 cores (2,10 GHz each).

[carlpett]

This is likely a duplicate of #89. Short version (it's a long thread) is that there is a bug/limit in Windows that causes these regular latency spikes, which are under some circumstances exaggerated.
One such circumstance reported is having mounted ISO files through iDRAC.

Unfortunately, this being a Windows bug, there's not much we can do aside from progressively replacing WMI.

[Mario-Hofstaetter]

@carlpett I am currently hitting (once again) this issue:

Collection timed out, still waiting for [service process]

I don't even need prometheus to scrape, opening http://localhost:9182/metrics in the browser shows the process and service metrics are missing.

There is a whole load of issues on this topic.. Is there a configureable timeout for the collector?

timeout-margin does not sound helpful:

windows_exporter/docs/example_config.yml

Line 13 in 6ba0297

timeout-margin: 0.5

as per README:

Seconds to subtract from the timeout allowed by the client. Tune to allow for overhead or high loads.

Couldn't find anything else?

EDIT:

Not sure if the exporter restart helped or my change,
but I added this to config.yaml

   collectors:
     enabled: "cpu,cs,logical_disk,memory,net,os,process,service,system,tcp,textfile,time"
++ maxScrapeDuration: 45s
   collector:
     service:

which didn't help ..... it was only successful once after start I see - collection is again timing out 😢
Is this setting working? It is not documented anywhere?

windows_exporter/collector/prometheus.go

Line 46 in 6ba0297

maxScrapeDuration time.Duration

[breed808]

Default timeout is 10 seconds, which can be changed by modifying the Prometheus scrape_timeout setting.

maxScrapeDuration is a red herring here, it's used by the exporter to respect the scrape timeout sent by Prometheus (via the X-Prometheus-Scrape-Timeout-Seconds header).

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.