[python-package] fix mypy error about cpu_count() methods #5786
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Contributes to #3756.
Contributes to #3867.
Fixes the following
mypy
errors.lightgbm
tries a few different approaches to determine the number of phyiscal/logical CPUs based on conditional imports. Adding type hints explicitly tellingmypy
that those methods will return integers resolves this issue.In this case,
mypy
caught a legitimate possible source of bugs 🎉If
joblib.cpu_count
can't be imported butpsutil.cpu_count
can,psutil.cpu_count()
is used to determine the number of available CPUs.According to the docs at https://psutil.readthedocs.io/en/latest/#psutil.cpu_count, that function returns
None
if it can't determine the number of CPUs. That's problematic because the result of that function is used unconditionally as if it was an integer:LightGBM/python-package/lightgbm/sklearn.py
Line 695 in 8811063
To deal with that, this PR proposes falling back to a value of
1
ifpsutil.cpu_count()
returnsNone
. That might mean that the user's physical resources are underutilized, but that's better than a runtime error.The other two methods can't return
None
?Right.
joblib.cpu_count()
callsloky.cpu_count()
(code link). I'm not 100% sure, but looks to me like likeloky.cpu_count()
cannot returnNone
based on the code paths at https://github.com/joblib/loky/blob/047d80623b7cf2d43500ed56ee0320b3e41c3f81/loky/backend/context.py#L77.multiprocessing.cpu_count()
raises aNotImplementedError
if it can't figure out the number of CPUs: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count.