[python-package] fix some warnings from mypy #3891

tara-jawahar · 2021-02-02T03:38:04Z

Fixed some mypy type warnings in setup.py, dask.py, and sklearn.py

ghost · 2021-02-02T03:38:17Z

All CLA requirements met.

jameslamb

Thanks very much for taking the time to contribute!

I left a few suggestions, most of which can be accepted directly in the browser.

After your next commit, can you please add a comment on this pull request with the output of running mypy --ignore-missing-imports python-package/?.

Also I hope you don't mind, but I've changed the title of this pull request to something that will be easy to understand in a changelog. We automatically generate changelogs from PR titles in this project: https://github.com/microsoft/LightGBM/releases/tag/untagged-e810435e8b97e053e761

python-package/lightgbm/dask.py

python-package/lightgbm/sklearn.py

python-package/setup.py

jameslamb · 2021-02-02T05:06:59Z

python-package/lightgbm/dask.py

@@ -27,7 +27,7 @@
 _PredictionDtype = Union[Type[np.float32], Type[np.float64], Type[np.int32], Type[np.int64]]


-def _find_open_port(worker_ip: str, local_listen_port: int, ports_to_skip: Iterable[int]) -> int:
+def _find_open_port(worker_ip: str, local_listen_port: int, ports_to_skip: Iterable[int]) -> Optional[int]:


Is this to fix this warning?

python-package/lightgbm/dask.py:71: error: Incompatible return value type (got "Optional[int]", expected "int")

I don't think this is the right fix. I think that mypy is (rightly) telling us that I wrote _find_open_port() in a way that makes it a little confusing to reason about the return type of out_port. Can you please change this back to int, and remove out_port = None on line 54? I think that should fix that warning.

I saw this conversation was marked resolved, but you did not implement what recommended.

Can you please change this back to int, and remove out_port = None on line 54, and see if it resolves the warning? If you disagree with that, can you please explain why?

tara-jawahar · 2021-02-03T02:08:34Z

Thank you for the comments!

Output of mypy --ignore-missing-imports python-package/ :

python-package/lightgbm/compat.py:14: error: Name 'pd_Series' already defined (possibly by an import)
python-package/lightgbm/compat.py:19: error: Name 'pd_DataFrame' already defined (possibly by an import)
python-package/lightgbm/compat.py:52: error: Name 'dt_DataTable' already defined on line 45
python-package/lightgbm/compat.py:128: error: Name 'dask_Array' already defined (possibly by an import)
python-package/lightgbm/compat.py:133: error: Name 'dask_DataFrame' already defined (possibly by an import)
python-package/lightgbm/compat.py:138: error: Name 'dask_Series' already defined (possibly by an import)
python-package/lightgbm/callback.py:28: error: First argument to namedtuple() should be 'CallbackEnv', not 'LightGBMCallbackEnv'
python-package/lightgbm/sklearn.py:795: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:795: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/sklearn.py:796: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:796: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/sklearn.py:868: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:868: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/sklearn.py:869: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:869: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/sklearn.py:991: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:991: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/sklearn.py:992: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:992: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/dask.py:284: error: Incompatible return value type (got "Dict[str, Any]", expected "LGBMModel")
python-package/lightgbm/dask.py:452: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:465: error: "_DaskLGBMModel" has no attribute "set_params"
python-package/lightgbm/dask.py:471: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:477: error: Item "_DaskLGBMModel" of "Union[_DaskLGBMModel, LGBMModel]" has no attribute "get_params"
python-package/lightgbm/dask.py:487: error: Signature of "fit" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:487: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:496: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMClassifier")
python-package/lightgbm/dask.py:512: error: Signature of "predict" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:512: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:523: error: Signature of "predict_proba" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:548: error: Signature of "fit" incompatible with supertype "LGBMRegressor"
python-package/lightgbm/dask.py:548: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:557: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRegressor")
python-package/lightgbm/dask.py:573: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:597: error: Signature of "fit" incompatible with supertype "LGBMRanker"
python-package/lightgbm/dask.py:597: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:611: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRanker")
python-package/lightgbm/dask.py:622: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:628: error: Signature of "predict" incompatible with supertype "LGBMModel"
Found 39 errors in 4 files (checked 10 source files)

jameslamb

Thanks very much! I left another round of suggestions.

python-package/lightgbm/sklearn.py

jameslamb · 2021-02-03T03:09:01Z

python-package/lightgbm/dask.py

@@ -564,7 +564,7 @@ def fit(
        )

    _base_doc = LGBMRegressor.fit.__doc__
-    _before_init_score, _init_score, _after_init_score = _base_doc.partition('init_score :')
+    _before_init_score, _init_score, _after_init_score = _base_doc.partition('init_score :') # type: ignore


ah, do these calls fix this?

python-package/lightgbm/dask.py:567: error: Item "None" of "Optional[str]" has no attribute "partition"

python-package/lightgbm/dask.py

jameslamb · 2021-02-03T03:12:13Z

python-package/lightgbm/dask.py

@@ -27,7 +27,7 @@
 _PredictionDtype = Union[Type[np.float32], Type[np.float64], Type[np.int32], Type[np.int64]]


-def _find_open_port(worker_ip: str, local_listen_port: int, ports_to_skip: Iterable[int]) -> int:
+def _find_open_port(worker_ip: str, local_listen_port: int, ports_to_skip: Iterable[int]) -> Optional[int]:


I saw this conversation was marked resolved, but you did not implement what recommended.

Can you please change this back to int, and remove out_port = None on line 54, and see if it resolves the warning? If you disagree with that, can you please explain why?

jameslamb · 2021-02-03T05:54:27Z

@tara-jawahar sorry for the inconvenience, but can you please merge the latest changes on the master branch into this? I just merged a fairly large change to the Dask module: #3883

…into mypy-errors

tara-jawahar · 2021-02-09T03:09:39Z

@jameslamb I selectively ignored some mypy errors based on your previous comments - let me know how they look! However, I was unable to fix the "Signature of [function in dask.py] incompatible with supertype LGBMClassifier/LGBMModel" errors.

jameslamb · 2021-02-09T03:16:00Z

Thanks very much! Can you please merge the latest changes from the master branch?

tara-jawahar · 2021-02-10T03:03:31Z

Hi James, sorry I missed your comment about that before! Thanks for pointing it out. Does it look like I did it correctly?

jameslamb · 2021-02-10T04:08:24Z

Hi James, sorry I missed your comment about that before! Thanks for pointing it out. Does it look like I did it correctly?

no problem! The merging does not look quite correct. We shouldn't see all these other commits from previous PRs.

I'd be happy to fix it if you'd like. Can I push to your branch?

tara-jawahar · 2021-02-11T00:26:21Z

Yes, go ahead!

jameslamb · 2021-02-11T02:09:19Z

Thanks! I just fixed this. In case you're curious, these are the commands I ran

git clone [email protected]:tara-jawahar/LightGBM.git tara-lgb
cd tara-lgb
git remote add upstream [email protected]:microsoft/LightGBM.git
git pull upstream master
git fetch origin mypy-errors
git checkout mypy-errors
git merge master
git push origin mypy-errors

python-package/lightgbm/callback.py

python-package/lightgbm/sklearn.py

jameslamb

This is looking pretty good, thanks for the help! Whenever you have time, could you answer my one new question about the changes you made to _doc elements in dask.py?

jameslamb · 2021-02-11T02:16:07Z

python-package/lightgbm/sklearn.py

-    _base_doc = (_base_doc[:_base_doc.find('group :')]
-                 + _base_doc[_base_doc.find('eval_set :'):])
+    _base_doc = (_base_doc[:_base_doc.find('group :')] # type: ignore
+                 + _base_doc[_base_doc.find('eval_set :'):]) # type: ignore


what issue did these # type: ignore comments on _base_doc fix? I ran mypy --ignore-missing-imports python-package/ on current master, and I don't see any notes related to these lines

python-package/lightgbm/dask.py:97: error: Incompatible return value type (got "Optional[int]", expected "int") python-package/lightgbm/dask.py:120: error: Need type annotation for 'lightgbm_ports' (hint: "lightgbm_ports: Set[<type>] = ...") python-package/lightgbm/dask.py:309: error: "Dict[str, Any]" has no attribute "status" python-package/lightgbm/dask.py:310: error: Incompatible return value type (got "Dict[str, Any]", expected "LGBMModel") python-package/lightgbm/dask.py:313: error: "Dict[str, Any]" has no attribute "key"; maybe "keys"? python-package/lightgbm/dask.py:478: error: "_DaskLGBMModel" has no attribute "_other_params" python-package/lightgbm/dask.py:496: error: "_DaskLGBMModel" has no attribute "get_params" python-package/lightgbm/dask.py:510: error: "_DaskLGBMModel" has no attribute "set_params" python-package/lightgbm/dask.py:516: error: "_DaskLGBMModel" has no attribute "get_params" python-package/lightgbm/dask.py:525: error: Item "_DaskLGBMModel" of "Union[_DaskLGBMModel, LGBMModel]" has no attribute "get_params" python-package/lightgbm/dask.py:587: error: Item "None" of "Optional[str]" has no attribute "partition" python-package/lightgbm/dask.py:602: error: Signature of "fit" incompatible with supertype "LGBMClassifier" python-package/lightgbm/dask.py:602: error: Signature of "fit" incompatible with supertype "LGBMModel" python-package/lightgbm/dask.py:610: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMClassifier") python-package/lightgbm/dask.py:636: error: Signature of "predict" incompatible with supertype "LGBMClassifier" python-package/lightgbm/dask.py:636: error: Signature of "predict" incompatible with supertype "LGBMModel" python-package/lightgbm/dask.py:654: error: Signature of "predict_proba" incompatible with supertype "LGBMClassifier" python-package/lightgbm/dask.py:738: error: Item "None" of "Optional[str]" has no attribute "partition" python-package/lightgbm/dask.py:753: error: Signature of "fit" incompatible with supertype "LGBMRegressor" python-package/lightgbm/dask.py:753: error: Signature of "fit" incompatible with supertype "LGBMModel" python-package/lightgbm/dask.py:761: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRegressor") python-package/lightgbm/dask.py:787: error: Signature of "predict" incompatible with supertype "LGBMModel" python-package/lightgbm/dask.py:870: error: Item "None" of "Optional[str]" has no attribute "partition" python-package/lightgbm/dask.py:885: error: Signature of "fit" incompatible with supertype "LGBMRanker" python-package/lightgbm/dask.py:885: error: Signature of "fit" incompatible with supertype "LGBMModel" python-package/lightgbm/dask.py:898: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRanker") python-package/lightgbm/dask.py:928: error: Signature of "predict" incompatible with supertype "LGBMModel"

these are the issues I was ignoring:

for this block of code:

Let me know if that's incorrect and I can take it out!

haha oh! I was looking the Dask module but clearly these changes and the warning come from sklearn.py. My mistake, sorry. Thanks for the explanation!

jameslamb

This is very close, thanks for the patience and explanations! If you accept the newest round of suggestions I left in the browser, I think tests will pass and this can be merged

python-package/lightgbm/callback.py

python-package/lightgbm/dask.py

python-package/lightgbm/sklearn.py

Co-authored-by: James Lamb <[email protected]>

tara-jawahar · 2021-02-14T23:02:01Z

Done! Thank you so much for the detailed comments - I'll keep all of these in mind.

jameslamb

thanks for the help! I hope you'll come back and contribute in the future.

github-actions · 2023-08-24T01:14:02Z

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

minor mypy type errors fixed

cd05e4c

tara-jawahar requested review from chivee, henry0312, jameslamb, StrikerRUS and wxchan as code owners February 2, 2021 03:38

jameslamb requested changes Feb 2, 2021

View reviewed changes

jameslamb changed the title ~~Issue#3867: minor mypy type warnings fixed~~ [python-package] fix some warnings from mypy Feb 2, 2021

jameslamb added the maintenance label Feb 2, 2021

fix some warnings from mypy

8232887

tara-jawahar force-pushed the mypy-errors branch from d1db63e to 8232887 Compare February 3, 2021 01:54

fix 3 mypy warnings

fa17ec1

jameslamb requested changes Feb 3, 2021

View reviewed changes

jameslamb mentioned this pull request Feb 6, 2021

[python-package] Some mypy fixes #3916

Merged

jameslamb added the in progress label Feb 7, 2021

tara-jawahar added 3 commits February 8, 2021 22:03

selectively ignored some mypy errors

ee68610

minor mypy type errors fixed

b87050d

Merge branch 'mypy-errors' of https://github.com/tara-jawahar/LightGBM …

eb8f8f7

…into mypy-errors

tara-jawahar added 3 commits February 9, 2021 21:49

minor mypy type errors fixed

2996216

minor mypy type errors fixed

638fefb

merge dask changes

5708735

tara-jawahar requested review from btrotta, guolinke, huanzhang12 and Laurae2 as code owners February 10, 2021 03:00

added import

6ad0743

jameslamb removed request for henry0312, chivee, huanzhang12, Laurae2, wxchan, guolinke and btrotta February 10, 2021 05:50

Merge branch 'master' into mypy-errors

ea68beb

jameslamb reviewed Feb 11, 2021

View reviewed changes

python-package/lightgbm/callback.py Outdated Show resolved Hide resolved

Update python-package/lightgbm/callback.py

ed5de61

jameslamb reviewed Feb 11, 2021

View reviewed changes

python-package/lightgbm/sklearn.py Outdated Show resolved Hide resolved

jameslamb reviewed Feb 11, 2021

View reviewed changes

python-package/lightgbm/sklearn.py Outdated Show resolved Hide resolved

Apply suggestions from code review

8bc4812

jameslamb requested changes Feb 11, 2021

View reviewed changes

jameslamb requested changes Feb 14, 2021

View reviewed changes

Apply suggestions from code review

a21dec3

Co-authored-by: James Lamb <[email protected]>

jameslamb approved these changes Feb 14, 2021

View reviewed changes

jameslamb merged commit eda1eff into microsoft:master Feb 15, 2021

StrikerRUS removed the in progress label Feb 16, 2021

github-actions bot locked as resolved and limited conversation to collaborators Aug 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[python-package] fix some warnings from mypy #3891

[python-package] fix some warnings from mypy #3891

tara-jawahar commented Feb 2, 2021

ghost commented Feb 2, 2021 •

edited by ghost

Loading

jameslamb left a comment

jameslamb Feb 2, 2021

jameslamb Feb 3, 2021

tara-jawahar commented Feb 3, 2021 •

edited

Loading

jameslamb left a comment

jameslamb Feb 3, 2021

jameslamb Feb 3, 2021

jameslamb commented Feb 3, 2021

tara-jawahar commented Feb 9, 2021 •

edited

Loading

jameslamb commented Feb 9, 2021

tara-jawahar commented Feb 10, 2021

jameslamb commented Feb 10, 2021

tara-jawahar commented Feb 11, 2021

jameslamb commented Feb 11, 2021

jameslamb left a comment

jameslamb Feb 11, 2021

tara-jawahar Feb 14, 2021 •

edited

Loading

jameslamb Feb 14, 2021

jameslamb left a comment

tara-jawahar commented Feb 14, 2021

jameslamb left a comment

github-actions bot commented Aug 24, 2023

[python-package] fix some warnings from mypy #3891

[python-package] fix some warnings from mypy #3891

Conversation

tara-jawahar commented Feb 2, 2021

ghost commented Feb 2, 2021 • edited by ghost Loading

jameslamb left a comment

Choose a reason for hiding this comment

jameslamb Feb 2, 2021

Choose a reason for hiding this comment

jameslamb Feb 3, 2021

Choose a reason for hiding this comment

tara-jawahar commented Feb 3, 2021 • edited Loading

jameslamb left a comment

Choose a reason for hiding this comment

jameslamb Feb 3, 2021

Choose a reason for hiding this comment

jameslamb Feb 3, 2021

Choose a reason for hiding this comment

jameslamb commented Feb 3, 2021

tara-jawahar commented Feb 9, 2021 • edited Loading

jameslamb commented Feb 9, 2021

tara-jawahar commented Feb 10, 2021

jameslamb commented Feb 10, 2021

tara-jawahar commented Feb 11, 2021

jameslamb commented Feb 11, 2021

jameslamb left a comment

Choose a reason for hiding this comment

jameslamb Feb 11, 2021

Choose a reason for hiding this comment

tara-jawahar Feb 14, 2021 • edited Loading

Choose a reason for hiding this comment

jameslamb Feb 14, 2021

Choose a reason for hiding this comment

jameslamb left a comment

Choose a reason for hiding this comment

tara-jawahar commented Feb 14, 2021

jameslamb left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 24, 2023

ghost commented Feb 2, 2021 •

edited by ghost

Loading

tara-jawahar commented Feb 3, 2021 •

edited

Loading

tara-jawahar commented Feb 9, 2021 •

edited

Loading

tara-jawahar Feb 14, 2021 •

edited

Loading