[Windows] Swap used is bigger than swap total

[thorstenkampe]

Summary

• OS: Windows 11
• Architecture: 64bit
• Psutil version: 5.9.0
• Python version: 3.10.2
• Type: core

Description

>>> import psutil
>>> psutil.swap_memory()
psutil._common.sswap(
    total=2147483648,
    used=4276985856,
    free=-2129502208,
    percent=199.2,
    sin=0,
    sout=0
)

[dbwiddis]

The negative value for free is probably a result of the calculation in #1927.

Root cause is probably that these numbers are fetched from GlobalMemoryStatusEx() which is documented as returning total and available page size values "for the system or the current process, whichever is smaller". Those docs state the system-wide values can be retrieved from GetPerformanceInfo().

I previously mentioned this in #1921 (comment)

[giampaolo]

Thanks @dbwiddis. Feel free to make a PR if you want. I'm not sure when I'll have time to get to this.

[rose112]

It seems that I have a similar issue using "Python version: 3.10.4 64Bit on Windows 11 pro 64Bit Version 21H2"
and psutil 5.9.0
The returned "total" value is correct, means identical to the size I set in the Windows settings.
So the values of "used", "free" and "percent" are incorrect. But the calculation of
"free = total - used" and "percent = 100 * ( used / total )" are both correct tly done.
My opinion is, that the internally received Value for "used" is wrong and so all related calculation will have invalid results.

[Danstiv]

Hello!
With swap disabled on Windows 10, I also get incorrect data on 5.9.2.
sswap(total=0, used=416116736, free=-416116736, percent=0.0, sin=0, sout=0)

[Danstiv]

This didn't solve my problem.
sswap(total=0, used=84897792, free=-84897792, percent=0.0, sin=0, sout=0)

[beqabeqa473]

Hello.

With swap be turned on, i am getting this result.

sswap(total=5100273664, used=5970190336, free=-869916672, percent=117.1, sin=0, sout=0)

I don't think this is information we are expecting to see.

[Danstiv]

More detailed and interesting example.

C:\Users\Danstiv>pip uninstall psutil
...
  Successfully uninstalled psutil-5.9.4

C:\Users\Danstiv>pip install git+https://github.com/giampaolo/psutil
...
Successfully installed psutil-5.9.4

C:\Users\Danstiv>python
...
>>> import psutil
>>> psutil.swap_memory()
sswap(total=0, used=0, free=0, percent=0.0, sin=0, sout=0)
>>> import webbrowser
>>> [webbrowser.open('https://youtube.com') for _ in range(10)]
[True, True, True, True, True, True, True, True, True, True]
>>> psutil.swap_memory()
sswap(total=0, used=38940672, free=-38940672, percent=0.0, sin=0, sout=0)

[dbwiddis]

Can you also show the physical memory values?

[Danstiv]

That's what I managed to find out, I hope I understand everything correctly.
In windows, there are different types of memory (cached, paged, nonpaged, kernel memory, etc.).
Now we are interested in 2 types, physical and committed.

Physical memory is actually available memory (if the computer has an 8 gb RAM stick installed, then there will be 8 gb of physical memory).
Information about physical memory can be obtained from the psutil.virtual memory function (it seems there is a terminology problem somewhere, but so far this is not very important).
psutil.virtual memory seems to correctly return physical memory usage information (at least on Windows) and no changes need to be made to this function.

Committed memory is the memory that Windows allocates to processes, it can be larger than physical memory because it is only allocated, not used.
In some cases, on the contrary, the committed memory may be less than the physical one, as I understand it, this happens when the process loads shared libraries (https://stackoverflow.com/questions/54715756/why-commit-size-is-somtimes-less-than-working-set-windows-memory).
That is, the committed memory of the process is not very large, but the physical memory, due to the libraries loaded into the process, becomes more than committed.
I don't quite understand how more physical memory can be occupied than committed in the entire system.
Probably, Windows has components that directly occupy physical memory without affecting the committed one (drivers, kernel, and similar low-level things).
In favor of this theory, I can say that I have never seen physical memory exceeding committed by more than a few hundred megabytes.

In addition, Windows has swap memory, it is used when physical memory becomes insufficient.
If swap is disabled, the maximum committed memory cannot be greater than the maximum physical memory.
If swap is enabled, the maximum committed memory may be greater than the physical one.
Maximum committed memory can be equal to physical memory + swap memory size.
For example, if the physical memory is 8 gb, and the swap size is 6 gb, then the maximum possible committed memory will be 14 gb.
If the swap is dynamic, and the initial size is 1 gb, and the maximum size is 10 gb, then as the memory fills up, the maximum possible committed memory will increase from 9 gb to 18 gb.

So, what does psutil.swap memory() do?
On Windows this function never seems to do what it's supposed to do, judging by the name.
In psutil 5.8.0, it returned committed memory (look at #807).
I get such results on my machine with 8 gb ram and disabled swap using 5.8.0.

>>> psutil.virtual_memory()
svmem(total=8474083328, available=3191390208, percent=62.3, used=5282693120, free=3191390208)
>>> psutil.swap_memory()
sswap(total=8474083328, used=5104918528, free=3369164800, percent=60.2, sin=0, sout=0)

And these are the results from another machine with 32 gb RAM and enabled swap.

>>> psutil.virtual_memory()
svmem(total=34200948736, available=18676355072, percent=45.4, used=15524593664, free=18676355072)
>>> psutil.swap_memory()
sswap(total=39301222400, used=21215404032, free=18085818368, percent=54.0, sin=0, sout=0)

Later, in 5.9.0 or earlier, this behavior was changed, and psutil.swap memory began to return incorrect results both with and without swap enabled (probably after mergeing pr #1927).
Also, some changes were made in 5.9.4, the result can be seen in the last comments in this issue (#2074).
Thus: psutil.swap memory continues to work incorrectly on Windows.

What I think should be done.
Let's start with the fact that in Windows, as I said at the beginning, there are several types of memory, but information about them is rarely required.
Memory is usually monitored in order to determine its shortage, and in time to take the necessary action.
And for these purposes, it is enough to track only 2 types of memory, physical and committed!

If physical memory runs out first and swap is turned off, Windows will physically not be able to allocate anything else, and something bad will happen.
If the committed memory runs out first (whether swap is enabled or not), Windows will also fail to allocate memory (this time due to software limitations), and something bad will happen again.
This is noticeable when using "VMware Workstation", if you activate the "Accelerate 3D graphics" option in the virtual machine settings and allocate a significant amount of memory for this (for example, 6 gb), then when the machine starts, these 6 gb do not get into physical memory, but are allocated Windows spending Committed memory.
And if suddenly committed memory runs out, despite the fact that there are a few more GB of physical memory, Windows can no longer function correctly, and various processes begin to cascade crash due to lack of memory.

Therefore, I suggest doing the following.
Option 1:
Return the behavior of psutil.swap memory as in 5.8.0.
This is not a good solution, because the returned data (committed memory info) will not match the function name (swap memory).
But this fix will allow to track committed memory again, as before.

Option 2:

1. Make a separate function for getting committed memory, which will return the same as swap memory in 5.8.0.
2. Fix psutil.swap memory to return information about swap.
   It can be calculated like this.
   If the maximum committed memory is equal to the maximum physical memory, then swap is disabled and zeros should be returned.
   Otherwise, the total swap size is total_swap_size - total_committed_size, and swap usage is max(0, committed_used - total_physical).

This solution is not good enough, since you need to add a separate function, test it, document it, etc., but it will finally solve all the problems with incorrect data that arise.

Correct me if I'm wrong somewhere, I hope this information will be useful to someone, and sorry for my English 🙂

[Danstiv]

Can you also show the physical memory values?

svmem(total=8474083328, available=3024613376, percent=64.3, used=5449469952, free=3024613376)

[dbwiddis]

Hey @Danstiv thanks for the detailed explanation. Let me try to add a bit more detail and suggestions here.

Option 1:
Return the behavior of psutil.swap memory as in 5.8.0.

This was broken, as the values being called "swap" were, in reality, "swap + physical". The behavior prior to my PR merged yesterday was based on MEMORYSTATUSEX and the numbers reported for "swap" were:

ullTotalPageFile

The current committed memory limit for the system or the current process, whichever is smaller, in bytes. To get the system-wide committed memory limit, call GetPerformanceInfo.

ullAvailPageFile

The maximum amount of memory the current process can commit, in bytes. This value is equal to or smaller than the system-wide available commit value. To calculate the system-wide available commit value, call GetPerformanceInfo and subtract the value of CommitTotal from the value of CommitLimit.

The problem with these numbers is that they are "for the system or the current process, whichever is smaller". The change in #2077 did precisely what these docs suggested for the "system-wide" value, calling GetPerformanceInfo and subtracting the value of CommitTotal from the value of CommitLimit:

totalSys = perfInfo.CommitLimit * pageSize;
availSys = totalSys - perfInfo.CommitTotal * pageSize;

But this fix will allow to track committed memory again, as before.

The values are there and can easily be recovered, if desired, by adding the physical and swap values. The psutil_virtual_mem() function where the snippet above comes from, returns four values:

• total physical memory
• available physical memory
• total system virtual memory (commit limit)
• available system virtual memory (commit limit - commit total)

And the calculation of the values psutil returns is:

mem = cext.virtual_mem()
totphys, availphys, totsys, availsys = mem

The physical memory part seems to be working. Here's the calculation for swap, which went in in #1927. mem[2] and mem[3] are the values you want:

mem = cext.virtual_mem()

total_phys = mem[0]
free_phys = mem[1]
total_system = mem[2]
free_system = mem[3]

# system memory (commit total/limit) is the sum of physical and swap
# thus physical memory values need to be substracted to get swap values
total = total_system - total_phys
free = min(total, free_system - free_phys)
used = total - free

Where you can simply back out the subtraction by adding total_phys back to total (thus commit limit exactly equals physical total, as expected). And for commit total, we can take the "free" value and add back in free_phys. So for the example above:

sswap(total=0, used=38940672, free=-38940672 ...

This indicates the min() calculation (-38940672) returned free_system - free_phys which is mem[3] - mem[1]. This is "(commit limit - commit total) - available physical memory". We know commit limit exactly matches physical total, so we can add that back in and deduce the commit total as "total - available physical memory + 38940672". So we have committed 37 MB more than we have used.

All this is a long way of saying "reverding to the old behavior gains us nothing".

Option 2:

Make a separate function for getting committed memory, which will return the same as swap memory in 5.8.0.

This is unneeded for the same reason. You can back into the calculation with the given numbers. However, the second part of option 2 is good:

Fix psutil.swap memory to return information about swap.
It can be calculated like this.
If the maximum committed memory is equal to the maximum physical memory, then swap is disabled and zeros should be returned.
Otherwise, the total swap size is total_swap_size - total_committed_size, and swap usage is max(0, committed_used - total_physical).

The "Otherwise" calculation is exactly what's being done, with the exception of the max() which also corresponds to the first statement of "zeros should be returned",

I think, ultimately, we just need to change this one line:

free = min(total, free_system - free_phys)

to

free = max(0, min(total, free_system - free_phys))

[dbwiddis]

Also, the root cause of this is the edge case where Commit Total is incremented before Physical Available is decremented:

CommitTotal

The number of pages currently committed by the system. Note that committing pages (using VirtualAlloc with MEM_COMMIT) changes this value immediately; however, the physical memory is not charged until the pages are accessed.

That instantaneous change to commit total can cause it to exceed physical "used" and in the edge case of "zero swap" results in the negative values we are seeing.

[giampaolo]

This bug aside, an important thing that we need and which is currently missing is a unit test which compares swap_memory() result with some system / CLI tool. E.g. on Linux we compare it against free CLI utility. I googled this a little, but it seems there's no CLI tool on Windows which returns info about swap memory. We currently unit-test physical memory usage via WMI / Win32_ComputerSystem:

psutil/psutil/tests/test_windows.py

Lines 121 to 124 in 55b588a

	def test_total_phymem(self):
	w = wmi.WMI().Win32_ComputerSystem()[0]
	self.assertEqual(int(w.TotalPhysicalMemory),
	psutil.virtual_memory().total)

Maybe we can do the same for swap mem by using WMI / Win32_PageFile? Is "pagefile" and "swap memory" the same thing on Windows? (I'm not sure)

[dbwiddis]

I'm working on another PR with this change to the free calculation and comments:

    # system memory (commit total/limit) is the sum of physical and swap
    # thus physical memory values need to be substracted to get swap values
    total = total_system - total_phys
    # commit total is incremented immediately (decrementing free_system)
    # while the corresponding free physical value is not decremented until
    # pages are accessed, so free_phys is an overestimate of the portion
    # free_system contributed to by physical memory, and in some edge cases
    # can exceed free system memory.
    free = max(0, free_system - free_phys)
    used = total - free

I'll submit as draft for now and look into the unit test options. There are also performance counters which may be relevant.

[dbwiddis]

[dbwiddis]

[Danstiv]

The values are there and can easily be recovered, if desired, by adding the physical and swap values.

Ok, that's cool.
In this case, really, no significant changes are required.

[dbwiddis]

This matches the performance counters above.

[dbwiddis]

Maybe we can do the same for swap mem by using WMI / Win32_PageFile?

Win32_PageFile appears deprecated and the Win32_PageFileUsage class is in units of pages (and I can't see an easy place to get page size). But the perfmon counters and associated WMI class look like a good alternative here. I think the commit limit minus the physical total should equal swap file size for a unit test. The committed bytes should be "close to" the commit total value.

Is "pagefile" and "swap memory" the same thing on Windows? (I'm not sure)

Yes, it is, but the way limits are handled are different between Windows and Linux.

Linux "guesses" whether there is swap available and never actually uses it all, using a "swappiness" property to control the level of "risk".

Windows enforces it strictly by reserving the worst case scenario and expanding the paging file (swap file) if there is any chance it will be needed.

Ultimately, though, the negative calculation is more of a human-readable interpretation separating the (accurate) commit total/limit vs. (accurate within a different context) physical total/used. Ultimately, "swap used" is an overestimate and "swap available" is an underestimate compared to a similar situation on Linux.

[giampaolo]

@dbwiddis FYI I took inspiration from your WMIC command line above and added a test for physical free mem, which was another important thing that was missing: 8e4099d.

[dbwiddis]

I added several more tests in the PR... may be out of order at this point after the merge...

[dbwiddis]

The values are there and can easily be recovered, if desired, by adding the physical and swap values.

Ok, that's cool. In this case, really, no significant changes are required.

Actually, @Danstiv, with the conditionals (min() and max()) we are now losing information.

In my java-based project I separately report "swap" values (using the perf counter percent calculation) and also report "virtual memory" values which are the sum of physical + swap and use the commit limit. Total virtual memory is actually what the operating system(s) use; it's transparent to userspace whether the memory being written to is on disk or in RAM, it's just "virtual memory".

So the "old values" were closer to this interpretation, but not consistent with the "swap file" calculations.

The swap file size is a somewhat accurate interpretation here, but the whole calculation breaks down when trying to use the "Commit Total" to calculate swap usage.... see this comment on my PR.

The performance counters do expose a "percent swap used" which I've added in the tests, and perhaps we should just multiply that fraction by the total swap size (which is an accurate calculation). But if we do that we lose the information in the "Commit Total" which includes reserved physical memory as well.

@giampaolo what are your thoughts here?

[giampaolo]

@giampaolo what are your thoughts here?

I must admit I am confused, also because it's a lot of information to process + I'm not good at math hehe.
With that said, it occurred to me that some years ago I worked on listing swap partitions in a separate branch which is still unmerged. In there I managed to list all page files (a path) and for each path get total and used metrics:

psutil/psutil/arch/windows/disk.c

Line 367 in 2974941

psutil_disk_swaps(PyObject *self, PyObject *args) {

This is a completely different approach but it seems to me that theoretically it could work.

[dbwiddis]

I must admit I am confused, also because it's a lot of information to process + I'm not good at math hehe.

I admit my spreadsheet has gotten a workout trying to resolve all this!

The key point here is the concept (Windows only) of committed (but not yet accessed) memory. With our current calculations we are assuming that is "used". This is an inconsistent interpretation with other operating systems.

With that said, it occurred to me that some years ago I worked on listing swap partitions in a separate branch which is still unmerged. In there I managed to list all page files (a path) and for each path get total and used metrics:

We do have access to percent usage per swap file if there are multiple ones, via WMI (see the test which assumes a single file, but we can filter by file name).

This is a completely different approach but it seems to me that theoretically it could work.

I agree here. The near-term question, however (for merging the PR), is whether to consider "committed" memory as used, or use WMI to calculate the (total) percent pagefile usage and report that, which loses the "commit total" value but is more consistent across OS's

[giampaolo]

The near-term question, however (for merging the PR), is whether to consider "committed" memory as used, or use WMI to calculate the (total) percent pagefile usage and report that, which loses the "commit total" value but is more consistent across OS's

Are you suggesting to infer used / free swap from Win32_PerfRawData_PerfOS_PagingFile -> PercentUsage?

Note: we cannot use the wmi python module, as that's a third-party dep used by unit-tests only.
If we want to use WMI (which would be a first in psutil core) we should either call it from C (best) or parse wmic command output (sub optimal, but probably still better than returning a negative value as we're doing now).

[dbwiddis]

Are you suggesting to infer used / free swap from Win32_PerfRawData_PerfOS_PagingFile -> PercentUsage?

Yes. I don't think it's available elsewhere.

Note: we cannot use the wmi python module, as that's a third-party dep used by unit-tests only.

Well, drat.

If we want to use WMI (which would be a first in psutil core) we should either call it from C (best) or parse wmic command output (sub optimal, but probably still better than returning a negative value as we're doing now).

We don't need to use WMI, it's just a convenient front-end to the performance counter api (PDH) which I've seen used elsewhere, e.g.:

psutil/psutil/arch/windows/wmi.c

Line 40 in 3398175

err = PdhGetFormattedCounterValue(

[giampaolo]

@dbwiddis take a look at #2161. With this I'm able to get the same values I get by running %windir%\system32\SystemPropertiesPerformance.exe, see https://superuser.com/a/858899.
I'd like to add a test case for total swap but I'm still unsure how.

[giampaolo]

@dbwiddis interesting comment in zabbix source code:
https://github.com/zabbix/zabbix/blob/32f4040712e35f1350b25a6403203825234dd770/src/libs/zbxsysinfo/win32/swap.c#L98-L130

	 * Due to the way Windows API functions report memory metrics,             *
	 * it is impossible to accurately retrieve swap (virtual memory)           *
	 * sizes as Windows reports the total commit memory which is physical      *
	 * memory plus swap file sizes. The only resolution that could be applied  *
	 * was calculatively deducting the swap sizes knowing the page and         *
	 * physical memory sizes.      

[dbwiddis]

Looks like we have competing PRs for the same information using completely different methods. I wonder how they compare to each other!

That comment you quoted looks like what we are already doing to calculate (total) swap size by taking the Commit Limit and subtracting physical memory. This part is working just fine.

The part that's not working is assuming the reserved portion of the "commit total" (used + reserved) is all on the swap file.

I've updated my PR with a counter-based approach using % Paging File. I'm confident in this approach as I use it on my own project and it seems to match other displays. (I'm less confident in my C and Python usage!)

It looks like your PR provides a different method but I see it may run afoul of permissions issues...