High memory usage on .16.0

[jesus-velez]

Hello!

I have a couple of servers reporting high memory usage by the windows_exporter. I am using the default configuration/collectors. Is there anyway to limit the resource usage on the exporter?

Exporter version:
windows_exporter, version 0.16.0 (branch: master, revision: f316d81d50738eb0410b0748c5dcdc6874afe95a) build user: appveyor-vm\appveyor@appveyor-vm build date: 20210225-10:52:19 go version: go1.15.6 platform: windows/amd64

OS Name: Microsoft Windows Server 2016 Standard
OS Version: 10.0.14393 N/A Build 14393

Handles NPM(K) PM(K) WS(K) CPU(s) Id SI ProcessName

---

1260 142 12748860 12496424 110,102.69 11388 0 windows_exporter

[breed808]

How long has the exporter been running on the affected hosts? When initially started, how much memory does the exporter use?

Graphing go_memstats_alloc_bytes, go_memstats_alloc_bytes_total and go_memstats_frees_total for this host should provide more info.

I'd also recommend navigating to http://hostname:9182/debug/pprof/heap and uploading the output here.

[jesus-velez]

Thanks for following up with me. The box has been up for 49 days so at least that much. I there any metric that records that information?

go_memstats_alloc_bytes

go_memstats_alloc_bytes_total

go_memstats_frees_total

heap.zip

[datamuc]

I have a similar thing, but we are on 0.15:
The go_memstat* things look pretty normal to me:

go_memstats.*bytes are very constant during the week:

go_memstats_alloc_bytes and frees_total:

What goes through the roof is windows_process_private_bytes:

And for what it's worth, pprof/heap and pprof/allocs:
pprof.zip

[jesus-velez]

I have both versions deployed and .15 also displays the same behavior. Whatever I can do to help please do let me know. @breed808.

[breed808]

Thanks all, this is really helpful info. I'll get through the heap/alloc dumps, and will try to reproduce the issue.

[datamuc]

It doesn't happen on every machine, but if it happens it seems to be the service collector that is leaking memory.

In general the windows_exporter is leaking memory anyways, it's just that it may leak with the service collector a huge amount of memory.

[jesus-velez]

@breed808, I don't know what @datamuc experience is but it seems like mostly servers with sql server installed get this issue. I have the exporter on application servers and database servers and I have yet to notice anything on the application servers. Another point to add is that I have seeing it on both VM and physical servers.

[breed808]

Unfortunately the heap dumps aren't too helpful: they're showing similar results to the go_memstats_alloc_bytes_total metric. @datamuc I wasn't able to view the dumps you provided, could you re-generate and upload them? Yours are ASCII text files, while the dump @jesus-velez submitted was gzip compressed data.

Are we able to confirm if the excessive memory consumption occurs when disabling all collectors that use WMI as a metric source? These are currently cpu_info, fsrmquota, msmq, mssql, remote_fx, service, system, terminal_services, and thermalzone

[datamuc]

Unfortunately the heap dumps aren't too helpful: they're showing similar results to the go_memstats_alloc_bytes_total metric. @datamuc I wasn't able to view the dumps you provided, could you re-generate and upload them? Yours are ASCII text files, while the dump @jesus-velez submitted was gzip compressed data.

Ok, I've accessed the urls with a browser that gave me the text files. Now I used Invoke-Webrequest and I got the binary stuff:

pprof.zip

I did some more tests, in any case there was an infinite powershell loop running that hit the /metrics endpoint.

In the zip there are 2 directories, host2 are the debug/pprof infos of a host that doesn't leak so much. I've done 2 snapshots in every test.

host1 is a host that has a pretty high leak, pprof1 and pprof2 where done with the following config:

collectors:
    enabled: "cpu,cs,logical_disk,net,os,system,textfile,memory,logon,tcp,terminal_services,thermalzone,iis,process"

and pprof3 and pprof4 where done with:

collectors:
    enabled: "cs,logical_disk,net,os,textfile,memory,logon,tcp,iis,process"

Are we able to confirm if the excessive memory consumption occurs when disabling all collectors that use WMI as a metric source? These are currently cpu_info, fsrmquota, msmq, mssql, remote_fx, service, system, terminal_services, and thermalzone

I've disabled all the mentioned collectors (pprof3/4) but it was still leaking.

windows_process_private_bytes while the endless loop was running:

And here without the loop, just normal prometheus scraping:

[breed808]

Hmm, those heap dumps are similar to those submitted earlier, with only a few MB allocated 😞

@datamuc from previous comments, it appears the service collector may be responsible, but host1 in your previous comment doesn't have it enabled?
It would be worth identifying the collectors responsible for the leaking. Are you free to do that?

I'll see if I can identify the commit or exporter version where the leaking was introduced.

[datamuc]

Hmm, those heap dumps are similar to those submitted earlier, with only a few MB allocated 😞

@datamuc from previous comments, it appears the service collector may be responsible, but host1 in your previous comment doesn't have it enabled?
It would be worth identifying the collectors responsible for the leaking. Are you free to do that?

I'll see if I can identify the commit or exporter version where the leaking was introduced.

Yes, we have deactivated the service collector globally because it contributed a lot to the leaks. It is better now, but still not good, so my guess is that it has something to do with the number of metrics returned? service collector has a lot of metrics.

I can do some more testing tomorrow I guess.

[datamuc]

The config:

log:
  level: fatal
collectors:
  enabled: "process"
collector:
  process:
    whitelist: "windows_exporter|exporter_exporter"
    blacklist: ""

And I did run this powershell script for an hour or so:

while($true) {
   $request = Invoke-Webrequest -URI http://127.77.1.1:5758/metrics
   $line = $request.RawContent -split "[`r`n]" | select-string -Pattern 'windows_process_private_bytes.*windows_exporter'
   ($line -split " ")[-1]
}

That resulted in the following graph of windows_process_private_bytes:

So, if it is a collector then process collector is one of them.

[breed808]

I've been able to reproduce this with the script. The memory leak is present in the process collector prior to 4f89133, so I'm inclined to believe it may be an exporter-wide leak that isn't specific to one collector.

Alternatively, it may be a leak on the Windows side when certain WMI and/or Perflib queries are submitted. The Go memory stats aren't showing any leaks, which isn't too helpful.

I'll continue testing to see if I can identify the commit that introduced the leak.

[datamuc]

I've did some more testing. If I only enable the textfile exporter, then there is no leak.

I tried to investigate a bit. I've seen that the windows_exporter uses a library from the perflib_exporter to access most of the performance_values. Maybe both implementations, the perflib and the old WMI one are leaking?

What follows maybe wrong, I've tried to reason about the code:

I've looked into what telegraf is doing, because it does a similar thing. They open pdh.dll and call functions from this library. Before they open a new query they close the old one?

The Close call leads to this code where they explicitly mention to free some memory.

Then I looked into the perflib_exporter, it comes with a perlfib package which provides access to the performance counters. Unlike telegraf they are not using pdh.dll because it is too high level?

Anyway, it does syscall.HKEY_PERFORMANCE_DATA to get the counters. So I googled a bit for it and found this page:

To obtain performance data from a remote system, call the RegConnectRegistry function. Use the computer name of the remote system and use HKEY_PERFORMANCE_DATA as the key. This call retrieves a key representing the performance data for the remote system. Use this key rather than HKEY_PERFORMANCE_DATA key to retrieve the data.

Be sure to use the RegCloseKey function to close the handle to the key when you are finished obtaining the performance data. This is important for both the local and remote cases:

• RegCloseKey(HKEY_PERFORMANCE_DATA) does not actually close a registry handle, but it clears all cached data and frees the loaded performance DLLs.
• RegCloseKey(hkeyRemotePerformanceData) closes the handle to the remote machine's registry.

But I cannot find the word "close" or "Close" anywhere in perflib_exporters codebase. So I thought, hooray, found the culprit and started a perflib_exporter which doesn't seem to leak at all. Long story short, I still have no clue what is going on. I think it is still possible that the perflib_exporter uses the perflib library a bit differently? Like reusing some datastructures while the windows_exporter is always asking for a new one or so?

[datamuc]

Does anybody have an idea? We are considering to move to telegraf and scrape that for the windows metrics.

[jesus-velez]

My priorities have shifted for the time being, but I have to come back to this issue pretty soon. I resorted to schedule a daily exporter bounce in the mean time. I can say that I found this issue has been around since the wmi exporter days. I recently logged into a server that we somehow missed on our exporter upgrades and the memory usage was really high.

[carlpett]

@datamuc That looks like a very good find, and even if the perflib exporter appears to not leak, I'd argue it it still incorrect not to close the key. It might require a bit of locking to do this right though, otherwise I think overlapping scrapes might lead to issues.
I'm not sure when I could commit to trying this out, but if anyone else wants to give it a go without the locking bits, I think we'd basically just need to add defer syscall.RegCloseKey(syscall.HKEY_PERFORMANCE_DATA) here.

[datamuc]

@carlpett I've added the mentioned line and compiled a binary. The change didn't help anything regarding the leak. 😞

[datamuc]

What do you think? Should we try to bring this issue up in the Prometheus & The Ecosystem Community Meeting? Maybe sombody there can help or knows somebody who can help?

[datamuc]

Are we able to confirm if the excessive memory consumption occurs when disabling all collectors that use WMI as a metric source? These are currently cpu_info, fsrmquota, msmq, mssql, remote_fx, service, system, terminal_services, and thermalzone

This is not true, other collectors also use wmi. I've started the exporter with

windows_exporter.exe --collectors.enabled=memory,os,net,time,cpu,cache

And this doesn't leak at all. So I'm pretty sure now that it leaks somewhere in github.com/StackExchange/wmi or github.com/go-ole/go-ole. The first one is definitly unmaintained, not so sure about the second one.

[breed808]

That's a good find! I'll see if I can identify which of those libraries are the cause of the leak.

[geraudster]

Hi, same problem here.
So I tried to build a new package with the commit of @datamuc on perflib and performed some tests (100 req/s on /metrics during 30 minutes) to compare both versions, the configured collectors are cpu,cs,logical_disk,net,os,system,textfile,process.

The patched version (the curve on the right) seems to do better than v0.16.0 (the one on the left) in terms of memory usage:

But a leak is still here. Do you have any other idea to fix this ?

Edit: the test was performed on master branch, so only the process collector was using StackExchange/wmi, which explains lower memory usage. After removing process collector, memory usage is constant around 20-30MB.

[datamuc]

Hi, same problem here. So I tried to build a new package with the commit of @datamuc on perflib and performed some tests (100 req/s on /metrics during 30 minutes) to compare both versions, the configured collectors are cpu,cs,logical_disk,net,os,system,textfile,process.

But a leak is still here. Do you have any other idea to fix this ?

We removed the process and service collector from our configuration (and added the tcp collector, so it is: [defaults] - service + tcp) now the memory usage is stable. It seems that every collector that makes use of github.com/StackExchange/wmi leaks.

[datamuc]

The patch above was merged into the perflib_exporter library with
leoluk/perflib_exporter#34.

Can sombody take care of updating the dependency in windows_exporter please?

[jesus-velez]

@datamuc Thanks for all the work that your are putting in on this.

[jesus-velez]

@datamuc

I found that telegraf was having a similar issue on Windows Server 2016. influxdata/telegraf#6807 (comment)

I don't know enough to be able to diagnose this myself, but looking at github.com/StackExchange/wmi, they are indeed using CoInitializeEx. What are you thoughts about this?

[datamuc]

Sounds promising to look deeper, but to be honest I have no idea. I was just lucky finding the leak in perflib_exporter, I have no idea of windows related programming at all, and just know a little bit go...

[audriuz]

Same issue keeps happening in our environment since implementation as well, latest windows_exporter version didn't solve it. Was not able to identify which exact software, feature or service may impact this happening on some random servers, but part of domain controllers impacted for sure (Windows Server 2016). Those servers are impacted heavily, check attachment.

It looks scary, but those gigabytes of memory mostly are taken not from physical RAM, but from virtual memory. As mentioned above in this conversation, seems that removing "process" and "service" collectors solves the issue or at least improving situation significantly. Even removing "service" collector alone, situation improves a lot, please check another attachment.

.

[YoByMe]

same problem here ... I have 2 servers with this problem.

any news ?

[breed808]

It's been a while since I've last looked at this, but I think disabling the WMI queries in the process collector by default may improve the situation.

Would anyone be able to test the branch in #998 to see if the process collector continues to experience memory leaks?

[exe-r]

Experiencing similar issue with leaks on latest 0.19.0, windows server 2016. Any permanent fix yet or should we disable process collector? @breed808 I think in my case #998 does not solve the issue cause i have two servers now, 1 which does not have leaks and one that does, collector.process.iis is enabled on both.

Have the following enabled on both

collectors:
  enabled: cpu,cs,logical_disk,net,os,service,system,memory,tcp,vmware,process,iis

The only difference is collector.process.whitelist, the affected server does have more processes included.

Non affected

    whitelist: "xService.?|windows_exporter.?"

Affected

whitelist: "xService.?|windows_exporter.?|app.+|antivirus.+|w3wp|Scheduler|xConnector|Ccm.+|xClient|inetinfo|.+agent|.+Agent"

I will try to remove few processes and see if it improves

Image of the leak

[breed808]

It's been some time since I last looked at this, but I believe my intention with #998 was to remove a potential leak in the process collector.
I suspect we have multiple leaks in each of the collectors. A fix for one won't resolve them all.

I've reopened #998 in #1062 if anyone would like to test the branch.

[exe-r]

It's been some time since I last looked at this, but I believe my intention with #998 was to remove a potential leak in the process collector. I suspect we have multiple leaks in each of the collectors. A fix for one won't resolve them all.

I've reopened #998 in #1062 if anyone would like to test the branch.

@breed808 Made a few tests and the most stable config was disabling scheduler_task collector completely. Its been 1 day since its running ok, will report back in case it changes. We only monitor 2 scheduled tasks so not sure why such aggressive leak is happening.

# working config
collectors:
  enabled: cpu,cs,logical_disk,net,os,service,system,memory,tcp,vmware,process,iis,netframework_clrexceptions
collector:
  process:
    whitelist: "xService.?|windows_exporter.?|app.+|antivirus.+|w3wp|Scheduler|xConnector|Ccm.+|xClient|inetinfo|.+agent|.+Agent"
  service:
    services-where: Name LIKE 'appname%'

[exe-r]

Coming back to the test @breed808. After 2 weeks having 6 servers running the above config I can confirm we stable at around 40-50mb~ of memory usage with no leaks. As soon as we enable scheduled_task collector the leak is very aggressive.

7 days usage example

[audriuz]

We're not using "scheduled_task" collector, but main problem with memory increase is "service" collector here. Only part of the servers are affected. Tried versions 0.18.1 and 0.19.0, same results. After removing "service" collector, huge difference, check the screenshot:

[breed808]

scheduled_task memory leak has been addressed in #1080. I'll try to investigate the service collector when there's time.

[breed808]

I've not been able to reproduce the service collector memory leak. Graph below shows resident memory with the service collector enabled with curl hitting the metrics endpoint constantly:

[kryztoval]

@breed808 I can't replicate it reliably but I am able to replicate it. I am seeing a bit of a trend where one of the instances is gettingmore and more memory as time passes, this started exactly at the problem when I added a rasberry pi as a prometheus server and targetted the windows_exporter with it to collect data every second. I am also hitting the same windows_exporter every second with a local instance of prometheus and a netdata collector on another raspberry pi. Having a single instance did not cause the memory leak but having both made it start happening. My guess would be there is something that is leaving the connection open or it is forcefully closed without letting the server clean up.

So for me to reproduce it I had to hit the windows_exporter more than 3 times per second at the very least.

[linuxlzj]

in windows 2016， i find use wmi query exist mem leak also

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.

[kryztoval]

This is not stale, the proccess leaks memory, specially and faster if it is being polled more than 3 times per second for whatever reason.

[datamuc]

Just an idea:

An option that might relieve the situation might be that the exporter has an option like --max-memory-consumption and monitors itself, and if it reaches that limit it just restarts itself or something like that.

[jkroepke]

Can someone provide some details?

Version of windows_exporter

Output of

• http://localhost:9182/debug/pprof/goroutine?debug=2
• http://localhost:9182/debug/pprof/heap?debug=1

as file attachment.

If possible, generate a trace:

curl -o trace.out http://localhost:9182/debug/pprof/trace?seconds=30

[datamuc]

We already had this here: #813 (comment)

But it doesn't help, because the memory is lost when interacting with Windows API somewhere. We found the leak in perflib. But there is at least another one in WMI. I can tell because the leakage stops if I disable all the collectors that make use of WMI:

#813 (comment)

[jkroepke]

@datamuc I saw there are pprof exists already exists.

However they coming from 0.16 and I would like to start at least form the lastest releases. In mean time, we included perflib libraries into windows_exporter and not longer depends on external contributors (https://github.com/prometheus-community/windows_exporter/tree/master/pkg/perflib)

[datamuc]

In mean time, we included perflib libraries into windows_exporter and not longer depends on external contributors (https://github.com/prometheus-community/windows_exporter/tree/master/pkg/perflib)

Interesting, I will try to come up with something next week.

[datamuc]

In mean time, we included perflib libraries into windows_exporter and not longer depends on external contributors (https://github.com/prometheus-community/windows_exporter/tree/master/pkg/perflib)

Interesting, I will try to come up with something next week.

Sorry, I don't have access to many windows machines anymore. The two that are left don't leak. So, if somebody still has leaks they have to do #813 (comment) themself.

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.