cpu_count is inconsistent between proc count and cpu count #614
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…or count on linux to make it match behavior on Windows.
|
Ping on this? (just got around to fixing the python 3 ci break related to this...) |
|
Confirmed this PR fixes the same bug I found. The current cpu_count_physical() returns number of sockets, not number of physical cores. |
|
I've just fixed this with a variant of your patch. Please confirm this works as expected. |
|
Works. Thanks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cpu_count() seems to return inconsistent results between platforms. I think it comes down to an assumption on whether logical=False means "physical cpus" or "physical cores". On Windows, it looks like it returns the core count, while Linux returns the cpu count.
In my case, I was assuming the former (cores), and that's the actual useful value I'm needing (in this case, I want to run something on N-1 cores, leaving one full core for the user). However, on Linux, with a dual-proc X5550, I'm getting 2 for logical=False, and 16 for logical=True. Launching 15 threads swamps the machine, and 2 is obviously not great :). It works as expected on Windows (logical=False gives 8).
I have a patch for _pslinux.py:cpu_count_physical to make it behave the way I expected, but I wanted to check if all of the above sounds right? I also haven't looked at any other platform beyond Linux and Windows, and what is happening there.