Missing windows_process_ metrics if multiple processes share the same name (java) - v0.30.0-rc.2

[aa-matthias]

Current Behavior

I do have multiple java processes running - before running the upgrade to 0.30.0-rc.2 I have received metrics for each process & pid. After upgrading only one java process is returned instead.

Expected Behavior

multiple processes sharing same process name should be exported.

Steps To Reproduce

No response

Environment

• windows_exporter Version: 0.30.0-rc.2
• Windows Server Version: 2019

windows_exporter logs

-

Anything else?

No response

[jkroepke]

@aa-matthias I remember your are having IIS as well?

I feel like I have the same issue, when a IIS site has been discarded and re-added.

[aa-matthias]

yes, IIS running here as well. but haven't discarded / readded any sites yet. config is simple here (proxy only).

[jkroepke]

Do you have any guidance? I have a VM with IIS but no plan

[aa-matthias]

🤔 this issue is not about IIS, is it? I tried the binaries of #1803 but the issue is still present here.

the metric that is only reporting one java process after running the upgrade is: windows_process_working_set_bytes - in 0.29.0 i got multiple processes reported correctly.

[jkroepke]

How do you try the binary from #1803?

[aa-matthias]

tried the artifacts from the action run: https://github.com/prometheus-community/windows_exporter/actions/runs/12138366080

[jkroepke]

Thats strange. I tested that with msedge and it works so far

# HELP windows_process_virtual_bytes Current size, in bytes, of the virtual address space that the process is using.
# TYPE windows_process_virtual_bytes gauge
windows_process_virtual_bytes{process="msedge",process_id="10092"} 3.515070754816e+12
windows_process_virtual_bytes{process="msedge",process_id="10236"} 2.306888421376e+12
windows_process_virtual_bytes{process="msedge",process_id="10936"} 2.342276800512e+12
windows_process_virtual_bytes{process="msedge",process_id="10980"} 2.307124219904e+12
windows_process_virtual_bytes{process="msedge",process_id="11236"} 3.513285238784e+12
windows_process_virtual_bytes{process="msedge",process_id="1188"} 3.513309761536e+12
windows_process_virtual_bytes{process="msedge",process_id="1276"} 3.513295818752e+12
windows_process_virtual_bytes{process="msedge",process_id="1496"} 2.306887766016e+12
windows_process_virtual_bytes{process="msedge",process_id="3628"} 2.272207331328e+12
windows_process_virtual_bytes{process="msedge",process_id="5644"} 3.513289699328e+12
windows_process_virtual_bytes{process="msedge",process_id="6380"} 3.515066908672e+12
windows_process_virtual_bytes{process="msedge",process_id="7148"} 3.51335469056e+12
windows_process_virtual_bytes{process="msedge",process_id="8124"} 2.306902740992e+12
windows_process_virtual_bytes{process="msedge",process_id="8768"} 2.306995408896e+12
windows_process_virtual_bytes{process="msedge",process_id="9908"} 3.513396461568e+12
# HELP windows_process_working_set_bytes Maximum number of bytes in the working set of this process at any point in time. The working set is the set of memory pages touched recently by the threads in the process.
# TYPE windows_process_working_set_bytes gauge
windows_process_working_set_bytes{process="msedge",process_id="10092"} 1.3238272e+07
windows_process_working_set_bytes{process="msedge",process_id="10236"} 1.7055744e+07
windows_process_working_set_bytes{process="msedge",process_id="10936"} 1.71671552e+08
windows_process_working_set_bytes{process="msedge",process_id="10980"} 1.47644416e+08
windows_process_working_set_bytes{process="msedge",process_id="11236"} 1.7268736e+07
windows_process_working_set_bytes{process="msedge",process_id="1188"} 1.25952e+07
windows_process_working_set_bytes{process="msedge",process_id="1276"} 4.0337408e+07
windows_process_working_set_bytes{process="msedge",process_id="1496"} 1.9943424e+07
windows_process_working_set_bytes{process="msedge",process_id="3628"} 8.11008e+06
windows_process_working_set_bytes{process="msedge",process_id="5644"} 3.9596032e+07
windows_process_working_set_bytes{process="msedge",process_id="6380"} 1.31137536e+08
windows_process_working_set_bytes{process="msedge",process_id="7148"} 1.08773376e+08
windows_process_working_set_bytes{process="msedge",process_id="8124"} 1.9992576e+07
windows_process_working_set_bytes{process="msedge",process_id="8768"} 3.0994432e+07
windows_process_working_set_bytes{process="msedge",process_id="9908"} 8.359936e+07

In that case, I need that output for cmd:

typeperf -qx | grep java | grep "Virtual Bytes"

[jkroepke]

I also take note that my fix in #1803 may fixes only an issue on Windows Server 2022 or higher.

[aa-matthias]

command output:

\Prozess(java)\Virtuelle Bytes (max.)
\Prozess(java#1)\Virtuelle Bytes (max.)
\Prozess(java#2)\Virtuelle Bytes (max.)
\Prozess(java#3)\Virtuelle Bytes (max.)
\Prozess(java#4)\Virtuelle Bytes (max.)
\Prozess(java)\Auslagerungsdatei (max. Bytes)
\Prozess(java#1)\Auslagerungsdatei (max. Bytes)
\Prozess(java#2)\Auslagerungsdatei (max. Bytes)
\Prozess(java#3)\Auslagerungsdatei (max. Bytes)
\Prozess(java#4)\Auslagerungsdatei (max. Bytes)
\Prozess(java)\Auslagerungsdatei (Bytes)
\Prozess(java#1)\Auslagerungsdatei (Bytes)
\Prozess(java#2)\Auslagerungsdatei (Bytes)
\Prozess(java#3)\Auslagerungsdatei (Bytes)
\Prozess(java#4)\Auslagerungsdatei (Bytes)
\Prozess(java)\Private Bytes
\Prozess(java#1)\Private Bytes
\Prozess(java#2)\Private Bytes
\Prozess(java#3)\Private Bytes
\Prozess(java#4)\Private Bytes
\Prozess(java)\Auslagerungsseiten (Bytes)
\Prozess(java#1)\Auslagerungsseiten (Bytes)
\Prozess(java#2)\Auslagerungsseiten (Bytes)
\Prozess(java#3)\Auslagerungsseiten (Bytes)
\Prozess(java#4)\Auslagerungsseiten (Bytes)
\Prozess(java)\Nicht-Auslagerungsseiten (Bytes)
\Prozess(java#1)\Nicht-Auslagerungsseiten (Bytes)
\Prozess(java#2)\Nicht-Auslagerungsseiten (Bytes)
\Prozess(java#3)\Nicht-Auslagerungsseiten (Bytes)
\Prozess(java#4)\Nicht-Auslagerungsseiten (Bytes)
\Prozess(java)\E/A-Bytes gelesen/s
\Prozess(java#1)\E/A-Bytes gelesen/s
\Prozess(java#2)\E/A-Bytes gelesen/s
\Prozess(java#3)\E/A-Bytes gelesen/s
\Prozess(java#4)\E/A-Bytes gelesen/s
\Prozess(java)\E/A-Bytes geschrieben/s
\Prozess(java#1)\E/A-Bytes geschrieben/s
\Prozess(java#2)\E/A-Bytes geschrieben/s
\Prozess(java#3)\E/A-Bytes geschrieben/s
\Prozess(java#4)\E/A-Bytes geschrieben/s
\Prozess(java)\Andere E/A-Bytes/s
\Prozess(java#1)\Andere E/A-Bytes/s
\Prozess(java#2)\Andere E/A-Bytes/s
\Prozess(java#3)\Andere E/A-Bytes/s
\Prozess(java#4)\Andere E/A-Bytes/s

windows_exporter scraped metrics:

windows_process_virtual_bytes{process="fontdrvhost",process_id="2136"} 2.203417976832e+12
windows_process_virtual_bytes{process="java",process_id="6204"} 1.303506944e+10
windows_process_virtual_bytes{process="lsass",process_id="748"} 2.203432185856e+12

[jkroepke]

Thats currently a "bigger" issue.

I always thought that Performance Index names are unique. They are unique as visible in our command output.

However, they are only unique through WMI and Powershell invoke WMI in background.

windows_exporter skips WMI and is directly asking the Windows API to gain performance counters. In that case the Index remains non unique.

While design the new collector, I assume the index is unique. But now, each process overwrite the statistics before.

I didn't mention that, because my dev box is Windows Server 2022 (and GitHub as well) And on Windows Server 2022, Microsoft fixed that issue on truly side, by making the index unique. They added the Process ID of the Server as suffix, making the index truly unique.

I have to think about an solution, but it's very important that you found that box.

[aa-matthias]

How is the process_id label generated then? Are metrics returned in order so a counter increment can be added by windows_exporter?

[jkroepke]

How is the process_id label generated then?

Its generated as metric from Performance counters. Technically, Process ID is just an metric like Virtual Bytes it is.

Are metrics returned in order so a counter increment can be added by windows_exporter?

It's undocumented. But I feel that maybe is not strong enough to depend against it.

[jkroepke]

So I identity a solution using the modern function, but it resulting into higher scrape times for the process collector (10x).

So for that case, I have re-introduce the old registry based collector for this, since it was solved in a performant way.

That guy had the same expericence: oshi/oshi#505 (comment)

However, as I mention it earlier, I have to switch away from key value pair to lists where the name is part of the value. That solves the deduplication issue as well.

But it will take time to refactor, but the plan is here.

[aa-matthias]

thanks keeping the issue updated. feel free to ping me for testing.