Wrong number of CPU cores detected in Windows Server multi Processor Group environment

[twisniewski78]

When running psutil.cpu_count(logical=False) I am getting 60 instead of 120 cores. Similarly when running psutil.cpu_percent(interval=1, percpu=True) I am getting utilization just for half of the CPU cores.
My system has two Processor Groups (each with 60 cores) and it seams that psutil is not processor group aware. On large systems processor resources are split into groups and it requires using special API to get correct numbers.

[giampaolo]

Sorry for such a big delay in responding. Can you provide more info about what kind of processors are these? Otherwise I really have no idea what kind of alternative API we could use.

[twisniewski78]

I have eight Xeons E7-2890 v2 with 15 cores each. My first guess would be NUMA architecture and Processor Group awareness rather then type of CPU. I was having the same issues on different system with 80 cores and different CPU model.

[giampaolo]

Looking back at this: what does len(psutil.cpu_times(percpu=True)) return? 60 or 120?

[giampaolo]

The problem may be here:

psutil/psutil/_psutil_windows.c

Line 582 in 6c3e3e1

if (ptr->Relationship == RelationProcessorCore)

As per:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms686694(v=vs.85).aspx
...if (ptr->Relationship == RelationNumaNode) we may have to increment the counter by 1

Still, that would not match len(psutil.cpu_percent(interval=1, percpu=True)) which according to your report would keep returning half the cores.

cpu_times(percpu=True) internally relies on dwNumberOfProcessors:

psutil/psutil/_psutil_windows.c

Line 980 in 6c3e3e1

for (i = 0; i < si.dwNumberOfProcessors; i++) {

...which should be "correct", so I'm really not sure what to think.

[twisniewski78]

Looking back at this: what does len(psutil.cpu_times(percpu=True)) return? 60 or 120?

60

[twisniewski78]

This is probably nothing new but maybe will give some different perspective: I see this issue in many programs. They are all processor groups unaware recognizing only group0. This is how NUMA nodes look like in my case: One can see there are eight NUMA nodes, each has one CPU with 15 cores. The nodes 0,1,4 and 5 are grouped into GROUP0, the remaining nodes make GROUP1. The phython.exe instance which I run psutil from was assigned by OS (or picked itself) GROUP1: This means it will be unable using group2 resources and that’s why it returned len(psutil.cpu_times(percpu=True)) 60 instead of 120 One workaround would be to start second python.exe process making sure it is assigned to another group and gather statistics including CPUs count from this process. To make it generic algorithm would have to “ 1. count total number of groups (because there may and will be more groups on other systems. Mine is only 50% populated, I will add two more groups and another 120 CPUs in the future) 2. for each group start separate process enforcing different and unique affinity (assignment to specific group) 3. compute in parallel statistics in all processes 4. gather and integrate data from distributed processes into a single instance for reporting 5. shut down all the processes except master one

[giampaolo]

Indeed MS doc states:

dwNumberOfProcessors
The number of logical processors in the current group. 
Note  For information about the physical processors shared by logical processors, 
call GetLogicalProcessorInformationEx with the RelationshipType parameter set to 
RelationProcessorPackage (3).

It looks like GetLogicalProcessorInformationEx should give us what we want:

When this function is called with a relationship type of 
RelationProcessorCore, it returns a PROCESSOR_RELATIONSHIP 
structure for every active processor core in every processor group 
in the system. 

[giampaolo]

Main problem is that I don't have the right hardware to test this against.

[giampaolo]

Note: os.cpu_count() on Python 3 relies on GetSystemInfo / dwNumberOfProcessors so it has the same problem.

[twisniewski78]

I know it may be inefficient, but I can run code for you on my system and send back the output if this helps. --- Main problem is that I don't have the right hardware to test this against.

[FJplant]

Is this still on-going?

…

Sent from my iPhone4s

On 02 Dec 2016, at 23:19, twisniewski78 ***@***.***> wrote: I know it may be inefficient, but I can run code for you on my system and send back the output if this helps. From: giampaolo ***@***.*** Sent: Friday, December 2, 2016 8:57 AM To: giampaolo/psutil ***@***.***> Cc: twisniewski78 ***@***.***>; Author ***@***.***> Subject: Re: [giampaolo/psutil] Wrong number of CPU cores detected in Windows Server multi Processor Group environment (#771) Main problem is that I don't have the right hardware to test this against. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#771 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AQeYyM_tjkw3Nx69xlAxcHp4WH0JwvRrks5rECOWgaJpZM4HbQUI> . <https://github.com/notifications/beacon/AQeYyNEPTEV-rq9uOv91PQoRiCrRjIJhks5rECOWgaJpZM4HbQUI.gif> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[giampaolo]

It is (still investigating and not fixed).

[FJplant]

Thank you Giampaolo.:-)

…

On Dec 3, 2016 10:55 PM, "giampaolo" ***@***.***> wrote: It is (still investigating and not fixed). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#771 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJSP__R6JtPJ7ZAXv-ZBw6PzYWkKomWYks5rEXS_gaJpZM4HbQUI> .

[waterkirin]

I may got the same issue. I have dual E5 2686 v3 with 18 cores each(36 cores and 72 logical processors ). When I ran psutil.cpu_count(logical = False) , I got 18. When I ran psutil.cpu_count(logical = True), i got 32. This is confusing.

[giampaolo]

Can you confirm os.cpu_count() has the same issue?

[giampaolo]

Here it is suggested that this may be due to 32 bit processes. Have you tried using Python 64 bit?

[FJplant]

Not completed... ^^

[giampaolo]

Another useful link:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms683194(v=vs.85).aspx

[robbuckley]

hi, i have the same issue. I am running 64 bit python 3.6.0 (anaconda) on 2 similar platforms
platform1 - 2x Xeon E5-2698v4. 20 cores/CPU = total 80 logical cpus with hyperthreading
platform2 - 2x Xeon E5-2697v3. 14 cores/CPU = total 56 logical cpus with hyperthreading

os.cpu_count(), multiprocessing.cpu_count() and psutil.cpu_count() all report the same thing: 40 cores on platform1 and 56 on platform2.

sysinternals coreinfo shows that platform1 has 2 processor groups and platform2 has 1 group.

i have a simple python program using a threadpool to launch multiple subprocess of an external tool which is entirely CPU bound. On platform 2 i can get it to 100% CPU for sustained periods. on Platform1 it tops out at ~50%

i would really like to be able to get this using 100% of platform1.

Im happy to run any test to help with this issue

[giampaolo]

@robbuckley can you confirm os.cpu_count() has the same issue? That has been introduced in Python 3.4. If you're on 2.7 you can use multiprocessing.cpu_count().

[robbuckley]

@giampaolo yes I confirm. I didn't realise they are the same. Im new to multiprocessing in python :-)

[giampaolo]

Could you please try Python 3.7 + os.cpu_count(). I suppose we should file a ticket on the Python bug tracker. Hopefully we can get some insight from there. The main problem here is that I don't have the right hardware to test this against.

[robbuckley]

How do I install python 3.7? according to https://www.python.org/dev/peps/pep-0537/ there doesnt seem to be an 3.7.0 alpha 1 yet, and i couldnt find anything in the conda channels including conda-forge.

I'm happy to help but dont have a lot of time to spend on it. Surely reporting a bug against the latest released version (3.6.1) should be OK?

[giampaolo]

Yeah, 3.6 is also OK.

[robbuckley]

@giampaolo is there anything more you need from me now?

[giampaolo]

Is os.cpu_count() on python 3.6 broken in the same manner?

[robbuckley]

yes, i confirm. See attached python session with os,cpu_count() reporting 40 cores, and output of microsoft coreinfo tool showing 80 logical processors in 2 processor groups

ci.txt
osc.txt

[giampaolo]

OK, so can you please file an issue on http://bugs.python.org/?

[robbuckley]

http://bugs.python.org/issue30581 raised

[giampaolo]

Also related: https://bugs.python.org/issue33166

[giampaolo]

I created a psutil branch using GetLogicalProcessorInformation (not GetLogicalProcessorInformationEx) to determine both logical and physical CPUs:
#1257
@twisniewski78 / @robbuckley : can you try it?
If you don't have Visual Studio I can send you a .whl file for Python 2.7 32 bit.

[twisniewski78]

Sadly I have no longer access to this environment or anything like it.

[giampaolo]

OK, this should finally be fixed as of #1257.

[giampaolo]

Closing out as fixed.