TYP: Assorted_Typing_Fix in pandas.io #25939

ryankarlos · 2019-03-31T05:13:52Z

closes Assorted Typing Fixups in pandas.io #25933
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

ryankarlos · 2019-03-31T05:15:39Z

There were some unused imports in packers module - i've left them in for now, not sure if the linter will complain as it was throwing warnings in flake8

codecov · 2019-03-31T05:36:21Z

Codecov Report

Merging #25939 into master will decrease coverage by 49.89%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #25939      +/-   ##
==========================================
- Coverage   91.81%   41.92%   -49.9%     
==========================================
  Files         175      175              
  Lines       52580    52597      +17     
==========================================
- Hits        48278    22049   -26229     
- Misses       4302    30548   +26246

Flag	Coverage Δ
#multiple	`?`
#single	`41.92% <100%> (-0.06%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/feather_format.py	`89.74% <100%> (+0.26%)`	⬆️
pandas/io/json/table_schema.py	`11.01% <100%> (-87.29%)`	⬇️
pandas/io/packers.py	`16.71% <100%> (-71.37%)`	⬇️
pandas/io/parquet.py	`20.19% <100%> (-64.28%)`	⬇️
pandas/io/json/normalize.py	`9.09% <100%> (-87.85%)`	⬇️
pandas/io/json/json.py	`63.51% <100%> (-29.72%)`	⬇️
pandas/io/html.py	`19.7% <100%> (-72.95%)`	⬇️
pandas/io/formats/latex.py	`0% <0%> (-100%)`	⬇️
pandas/io/sas/sas_constants.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/plotting.py	`0% <0%> (-100%)`	⬇️
... and 137 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update de3a85c...f24159e. Read the comment docs.

codecov · 2019-03-31T05:36:21Z

Codecov Report

Merging #25939 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #25939      +/-   ##
==========================================
- Coverage    91.9%   91.89%   -0.01%     
==========================================
  Files         175      175              
  Lines       52485    52485              
==========================================
- Hits        48235    48230       -5     
- Misses       4250     4255       +5

Flag	Coverage Δ
#multiple	`90.45% <ø> (ø)`	⬆️
#single	`40.75% <ø> (-0.13%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	`75% <0%> (-12.5%)`	⬇️
pandas/core/frame.py	`96.79% <0%> (-0.12%)`	⬇️
pandas/util/testing.py	`90.62% <0%> (-0.11%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d9b702...e08ab93. Read the comment docs.

ryankarlos · 2019-03-31T14:46:37Z

@WillAyd Was getting an azure pipeline error when removing ununsed import DataFrame, Panel, Float64Index, Int64Index but removing them makes everything fail now

WillAyd · 2019-03-31T15:23:04Z

@ryankarlos that does seem strange - have you been able to debug that issue locally?

ryankarlos · 2019-03-31T23:20:54Z

@WillAyd Something strange - when i debug locally I can reproduce the same error - problem seems to be mainly with packers - when i include the unused imports DataFrame, Panel in packers.py and run the test on test_packers.py it passes but when I exclude either of them or both from the packers module it fails. I am not too experienced with the import machinery but this seems a bit strange. There are a few more unused imports like Float64Index, Int64Index which do not seem to be causing the tests to fail when removing them. Is there a way to make the linter ignore unused imports (in which case I could just include them all) or is it critical that these are picked up ?

TEST 1 (after including both DataFrame and Panel imports )

====================================================================
test session starts ====================================================================
platform darwin -- Python 3.7.2, pytest-4.1.1, py-1.7.0, pluggy-0.8.1`
hypothesis profile 'ci' -> deadline=500, timeout=unlimited, suppress_health_check=[HealthCheck.too_slow], database=DirectoryBasedExampleDatabase('/Users/ryannazareth/Documents/Python_sprints/pandas-ryankarlos/.hypothesis/examples')
rootdir: /Users/ryannazareth/Documents/Python_sprints/pandas-ryankarlos, inifile: setup.cfg
plugins: xdist-1.26.0, forked-0.2, cov-2.6.1, hypothesis-4.5.0
collected 83 items

pandas/tests/io/test_packers.py .........................................s.s.ss.s............s.s..s.s.sss.......... [100%]

=========================================================== 71 passed, 12 skipped in 2.38 seconds ===========================================================

TEST 2 (after excluding Panel import)

            return make_block(values=values,
                              klass=getattr(internals, b['klass']),
                              placement=placement,
                              dtype=b['dtype'])

        blocks = [create_block(b) for b in obj['blocks']]

      return globals()[obj['klass']](BlockManager(blocks, axes))

E KeyError: 'Panel'

pandas/io/packers.py:667: KeyError
===================================================== 22 failed, 49 passed, 12 skipped in 6.31 seconds ======================================================

jreback · 2019-03-31T23:36:43Z

msgpack needs the imports to correctly dessrialize things

you cannot remove them

ryankarlos · 2019-04-01T20:09:10Z

Thanks, ok just pushed the imports but what do i do with the linter complaining about unused imports ?

jbrockmendel · 2019-04-08T18:25:40Z

but what do i do with the linter complaining about unused imports ?

# noqa:F401 I think

Can you merge master

ryankarlos · 2019-04-09T23:14:38Z

@jbrockmendel thanks

…a,io PRs #25936 #25940 #25939 (#26019)

WillAyd

lgtm @jreback

jreback · 2019-04-10T12:58:27Z

pandas/io/json/json.py

@@ -12,8 +12,9 @@

 from pandas.core.dtypes.common import is_period_dtype

-from pandas import DataFrame, MultiIndex, Series, compat, isna, to_datetime
-from pandas.core.reshape.concat import concat
+from pandas import MultiIndex, compat, concat, isna, to_datetime


DataFrame/Series should be from pandas (same for all places)

WillAyd · 2019-04-10T13:02:38Z

@ryankarlos do we even need this PR any more? I think fixed by the implicit import PR you did earlier

ryankarlos

@WillAyd I still think it would be useful to have this PR as Ive just noticed a few places which could still be importing from top level even after reverting all changes ... see comments below.
I fixed these now in latest commit

ryankarlos · 2019-04-10T16:37:56Z

pandas/io/json/json.py

+from pandas import compat
+from pandas.core.frame import DataFrame
+from pandas.core.index import MultiIndex
+from pandas.core.missing import isna
 from pandas.core.reshape.concat import concat


ryankarlos · 2019-04-10T16:38:17Z

pandas/io/json/table_schema.py

@@ -12,9 +12,9 @@
    is_datetime64tz_dtype, is_integer_dtype, is_numeric_dtype, is_period_dtype,
    is_string_dtype, is_timedelta64_dtype)

-from pandas import DataFrame
 from pandas.api.types import CategoricalDtype


@WillAyd and here

WillAyd · 2019-04-10T17:00:12Z

I would rather just boil this PR down to removing the blacklist items if no further changes to imports are required

ryankarlos · 2019-04-10T17:58:50Z

So OK for me to completely reset changes in this PR as well and only keep mypy.ini changes ?

WillAyd · 2019-04-10T17:59:54Z

Yes - the rest of the elements are arguably orthogonal at this point. Can be changed in separate more comprehensive PRs if we desire

ryankarlos · 2019-04-11T00:00:00Z

Ok thanks - just pushed only mypy changes and reset everything else here and in #25936. Just waiting for tests to pass.

WillAyd · 2019-04-11T00:47:56Z

Thanks @ryankarlos !

WillAyd added the Typing type annotations, mypy/pyright type checking label Apr 1, 2019

ryankarlos mentioned this pull request Apr 7, 2019

TYP: Replace wildcardimports in toplevelinit as precursor for reshape,stata,io PRs #25936 #25940 #25939 #26019

Merged

4 tasks

jreback pushed a commit that referenced this pull request Apr 10, 2019

Replace wildcardimports in toplevelinit as precursor for reshape,stat…

9d66bdf

…a,io PRs #25936 #25940 #25939 (#26019)

WillAyd approved these changes Apr 10, 2019

View reviewed changes

WillAyd added this to the 0.25.0 milestone Apr 10, 2019

jreback requested changes Apr 10, 2019

View reviewed changes

ryankarlos commented Apr 10, 2019

View reviewed changes

ryankarlos force-pushed the Assorted_Typing_Fix_pandas.io branch from e0097f9 to 90b4383 Compare April 10, 2019 17:41

Reset PR and only add mypy.ini changes

e08ab93

ryankarlos force-pushed the Assorted_Typing_Fix_pandas.io branch from 90b4383 to e08ab93 Compare April 10, 2019 23:45

WillAyd merged commit e02ec8f into pandas-dev:master Apr 11, 2019

WillAyd mentioned this pull request Apr 11, 2019

Typing Cleanup - Remove Blacklisted Items #25882

Closed

65 tasks

ryankarlos deleted the Assorted_Typing_Fix_pandas.io branch April 11, 2019 21:08

ryankarlos changed the title ~~Assorted_Typing_Fix in pandas.io~~ TYP:Assorted_Typing_Fix in pandas.io Jan 6, 2020

ryankarlos changed the title ~~TYP:Assorted_Typing_Fix in pandas.io~~ TYP: Assorted_Typing_Fix in pandas.io Jan 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TYP: Assorted_Typing_Fix in pandas.io #25939

TYP: Assorted_Typing_Fix in pandas.io #25939

ryankarlos commented Mar 31, 2019

ryankarlos commented Mar 31, 2019

codecov bot commented Mar 31, 2019

codecov bot commented Mar 31, 2019 •

edited

Loading

ryankarlos commented Mar 31, 2019

WillAyd commented Mar 31, 2019

ryankarlos commented Mar 31, 2019 •

edited

Loading

jreback commented Mar 31, 2019

ryankarlos commented Apr 1, 2019 •

edited

Loading

jbrockmendel commented Apr 8, 2019

ryankarlos commented Apr 9, 2019

WillAyd left a comment

jreback Apr 10, 2019

WillAyd commented Apr 10, 2019

ryankarlos left a comment •

edited

Loading

ryankarlos Apr 10, 2019 •

edited

Loading

ryankarlos Apr 10, 2019

WillAyd commented Apr 10, 2019

ryankarlos commented Apr 10, 2019

WillAyd commented Apr 10, 2019

ryankarlos commented Apr 11, 2019

WillAyd commented Apr 11, 2019

TYP: Assorted_Typing_Fix in pandas.io #25939

TYP: Assorted_Typing_Fix in pandas.io #25939

Conversation

ryankarlos commented Mar 31, 2019

ryankarlos commented Mar 31, 2019

codecov bot commented Mar 31, 2019

Codecov Report

codecov bot commented Mar 31, 2019 • edited Loading

Codecov Report

ryankarlos commented Mar 31, 2019

WillAyd commented Mar 31, 2019

ryankarlos commented Mar 31, 2019 • edited Loading

jreback commented Mar 31, 2019

ryankarlos commented Apr 1, 2019 • edited Loading

jbrockmendel commented Apr 8, 2019

ryankarlos commented Apr 9, 2019

WillAyd left a comment

Choose a reason for hiding this comment

jreback Apr 10, 2019

Choose a reason for hiding this comment

WillAyd commented Apr 10, 2019

ryankarlos left a comment • edited Loading

Choose a reason for hiding this comment

ryankarlos Apr 10, 2019 • edited Loading

Choose a reason for hiding this comment

ryankarlos Apr 10, 2019

Choose a reason for hiding this comment

WillAyd commented Apr 10, 2019

ryankarlos commented Apr 10, 2019

WillAyd commented Apr 10, 2019

ryankarlos commented Apr 11, 2019

WillAyd commented Apr 11, 2019

codecov bot commented Mar 31, 2019 •

edited

Loading

ryankarlos commented Mar 31, 2019 •

edited

Loading

ryankarlos commented Apr 1, 2019 •

edited

Loading

ryankarlos left a comment •

edited

Loading

ryankarlos Apr 10, 2019 •

edited

Loading