Initial pandas.typing Module #25884

WillAyd · 2019-03-26T19:18:11Z

ref #25791 maybe more of a conversation starter than complete PR but wanted to stub out some types that appear commonly in pandas code-base.

Haven't used these yet as wanted to agree on type first. I wouldn't be surprised if we start seeing some issues pop up that would require real development changes once these get utilized

@jreback @jbrockmendel

jreback · 2019-03-26T19:29:20Z

cool, can you use these in a couple of places? and some examples / docs-strings?

codecov · 2019-03-26T19:56:15Z

Codecov Report

Merging #25884 into master will decrease coverage by 0.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master   #25884      +/-   ##
==========================================
- Coverage   91.48%   91.46%   -0.02%     
==========================================
  Files         175      176       +1     
  Lines       52885    52892       +7     
==========================================
  Hits        48380    48380              
- Misses       4505     4512       +7

Flag	Coverage Δ
#multiple	`90.03% <0%> (-0.02%)`	⬇️
#single	`41.82% <0%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/typing.py	`0% <0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d404460...e230500. Read the comment docs.

codecov · 2019-03-26T19:56:29Z

Codecov Report

Merging #25884 into master will decrease coverage by 0.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master   #25884      +/-   ##
==========================================
- Coverage   91.48%   91.46%   -0.02%     
==========================================
  Files         175      176       +1     
  Lines       52885    52892       +7     
==========================================
  Hits        48380    48380              
- Misses       4505     4512       +7

Flag	Coverage Δ
#multiple	`90.03% <0%> (-0.02%)`	⬇️
#single	`41.82% <0%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/typing.py	`0% <0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d404460...e230500. Read the comment docs.

codecov · 2019-03-26T19:56:49Z

Codecov Report

Merging #25884 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #25884      +/-   ##
==========================================
- Coverage    91.8%   91.79%   -0.01%     
==========================================
  Files         174      175       +1     
  Lines       52536    52540       +4     
==========================================
  Hits        48231    48231              
- Misses       4305     4309       +4

Flag	Coverage Δ
#multiple	`90.35% <100%> (ø)`	⬆️
#single	`41.88% <83.33%> (-0.07%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/gcs.py	`80% <100%> (ø)`	⬆️
pandas/io/parsers.py	`95.69% <100%> (ø)`	⬆️
pandas/io/s3.py	`89.47% <100%> (ø)`	⬆️
pandas/_typing.py	`100% <100%> (ø)`
pandas/io/gbq.py	`75% <0%> (-12.5%)`	⬇️
pandas/core/frame.py	`96.79% <0%> (-0.12%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d4c89f...7e31345. Read the comment docs.

jbrockmendel · 2019-03-27T00:05:55Z

pandas/typing.py

+from pandas._libs.tslibs import Period, NaT, Timedelta, Timestamp
+
+
+DateTimeLike = Union[datetime, timedelta, Period, Timedelta, Timestamp]


Do/should the numpy versions go in here?

WillAyd · 2019-03-27T01:51:35Z

I don’t think so but not sure I follow - what’s the link between NumPy version and types?

…

Sent from my iPhone

On Mar 26, 2019, at 5:05 PM, jbrockmendel ***@***.***> wrote: @jbrockmendel commented on this pull request. In pandas/typing.py: > @@ -0,0 +1,14 @@ +from datetime import datetime, timedelta +from pathlib import Path +from typing import AnyStr, IO, Union + +from pandas._libs.tslibs import Period, NaT, Timedelta, Timestamp + + +DateTimeLike = Union[datetime, timedelta, Period, Timedelta, Timestamp] Do/should the numpy versions go in here? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

jbrockmendel · 2019-03-27T02:10:37Z

I don’t think so but not sure I follow - what’s the link between NumPy version and types?

Bad phrasing on my part. I was referring to np.datetime64 and np.timedelta64 objects. Many of our functions that accept datetime inputs will also accept np.datetime64 inputs. Ditto for timedelta.

WillAyd · 2019-03-27T03:59:25Z

Sure just added the FilePathOrBuffer type to io.parsers. CI will fail with the following:

pandas/io/parsers.py:445: error: Item "str" of "Union[str, Path, IO[Any]]" has no attribute "close"
pandas/io/parsers.py:445: error: Item "Path" of "Union[str, Path, IO[Any]]" has no attribute "close"

There's an implicit assumption in the codebase that a local variable should_close dictates whether or not a close method gets called, which Mypy doesn't like. I suppose that is rather fragile so refactor could be in line

@jbrockmendel any thoughts on where best to try out the DateTimeLike types?

jbrockmendel · 2019-03-27T15:56:38Z

any thoughts on where best to try out the DateTimeLike types?

Not really. I guess I'm not clear on the use case. Aside from running mypy, are we actually doing anything with these?

WillAyd · 2019-03-27T16:00:49Z

Use case would be to improve readability and maintainability of code base and in theory reducing bugs. There's some refactor that will go hand in hand with it (as you might have seen with some of the IO stuff above).

This is just starting things off and there's a long way to go. I think I might split off the datetimelike stuff for this for now and just focus on IO. Can see how that goes and learn from there

WillAyd · 2019-03-28T01:58:50Z

pandas/io/parsers.py

+    # Union[FilePathOrBuffer, s3fs.S3File, gcsfs.GCSFile]
+    # though mypy handling of conditional imports is difficult.
+    # See https://github.com/python/mypy/issues/1297
+    fp_or_buf, _, compression, should_close = get_filepath_or_buffer(


It's mentioned in the comments but I changed the variable name here from filepath_or_buffer to fp_or_buf to intentionally NOT shadow the parameter from the signature.

As mentioned in the comment this local variable could potentially introduce new types for S3 and GCP and I don't think there is a great way with typing to statically analyze conditional imports like those just yet, so it's a clearer delimitation IMO to assign the return of this function to a separate variable

jreback · 2019-03-28T20:42:48Z

see some suggestions: #25829 (can be a followup)

jreback · 2019-03-28T20:44:00Z

looks fine. better to start small. merge if you like (or add more up 2 you)

jorisvandenbossche · 2019-03-28T21:52:39Z

doc/source/whatsnew/v0.25.0.rst

+Type Hints and ``pandas.typing``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In accordance with :pep:`484` pandas has introduced Type Hints and a new ``pandas.typing`` module containing aliases for idiomatic pandas types into the code base. We will be continually adding annotations to the code base to improve readability, reduce code maintenance and proactively identify bugs.


Do we make this public?
(which mean: maintaining compatibility) Maybe at least not initially?

Yea I don't see a downside to making private. Can do that on next iteration

OK just made this private and updated documentation - lmk what you think

jreback · 2019-03-29T12:09:50Z

doc/source/whatsnew/v0.25.0.rst

+
+.. _whatsnew_0250.enhancements.typing:
+
+Type Hints and ``pandas.typing``


i would remove this entirely. if its private it is not to be relied upon.

jreback

needs a tests as indicated, other lgtm. ping on green.

jreback · 2019-03-30T18:56:27Z

pandas/_typing.py

@@ -0,0 +1,4 @@
+from pathlib import Path


can you add a test that asserts this in pandas/tests/types/test_api.py

Yea no problem. So the existing test(s) exclude any privately named modules:

pandas/pandas/tests/api/test_api.py

Line 14 in 2792705

result = sorted(f for f in dir(namespace) if not f.startswith('_'))

Are you asking to revisit that logic or simply add a test with this as an exception to make sure it lives there?

oh i c. nvm then. though I think we should actually check and lock down all modules (so add back the private ones). in a new PR / issue though.

jreback · 2019-03-30T19:12:49Z

merging this so can start using it. less concerned about the test as its private.

Initial typing module

e230500

WillAyd added the Typing type annotations, mypy/pyright type checking label Mar 26, 2019

jbrockmendel reviewed Mar 27, 2019

View reviewed changes

Added FilePathOrBuffer type to io.parsers

49c0c69

WillAyd added 2 commits March 26, 2019 21:13

Added FilePathOrBuffer to io.common

766ad86

Added cast to suppress mypy errors

8f4973c

WillAyd added 8 commits March 27, 2019 09:57

Removed DateTimeLike types

b621f32

Merge remote-tracking branch 'upstream/master' into pandas-typing

3fd6814

Removed type hints from pandas.io.common

6b9f0eb

Removed local variable shadowing, refactored for new types

b17ad46

Doc note for pandas.typing

ed3fc58

Added typing to exposed API test

34c00bf

Merge remote-tracking branch 'upstream/master' into pandas-typing

6044072

isort fixup

3daf02f

WillAyd commented Mar 28, 2019

View reviewed changes

jreback added this to the 0.25.0 milestone Mar 28, 2019

jorisvandenbossche reviewed Mar 28, 2019

View reviewed changes

WillAyd added 2 commits March 28, 2019 15:54

Merge remote-tracking branch 'upstream/master' into pandas-typing

e0fb927

Privated typing module

3ff41d3

WillAyd added 3 commits March 28, 2019 15:58

Updated docs

133e045

Privatized typing in test_api

688f705

Reset changes to test_api

7bd5a62

jreback requested changes Mar 29, 2019

View reviewed changes

WillAyd added 2 commits March 29, 2019 09:44

Merge remote-tracking branch 'upstream/master' into pandas-typing

fea9765

Reverted docstring changes

7e31345

jreback requested changes Mar 30, 2019

View reviewed changes

jreback approved these changes Mar 30, 2019

View reviewed changes

jreback merged commit e1aa6f3 into pandas-dev:master Mar 30, 2019

WillAyd deleted the pandas-typing branch March 30, 2019 19:12

This was referenced Mar 30, 2019

Add Test to Explicitly Check Private Modules Exposed via API #25930

Closed

Introduce pandas.typing #25791

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial pandas.typing Module #25884

Initial pandas.typing Module #25884

WillAyd commented Mar 26, 2019

jreback commented Mar 26, 2019

codecov bot commented Mar 26, 2019

codecov bot commented Mar 26, 2019

codecov bot commented Mar 26, 2019 •

edited

Loading

jbrockmendel Mar 27, 2019

WillAyd commented Mar 27, 2019 via email

jbrockmendel commented Mar 27, 2019

WillAyd commented Mar 27, 2019 •

edited

Loading

jbrockmendel commented Mar 27, 2019

WillAyd commented Mar 27, 2019

WillAyd Mar 28, 2019

jreback commented Mar 28, 2019

jreback commented Mar 28, 2019

jorisvandenbossche Mar 28, 2019

WillAyd Mar 28, 2019

WillAyd Mar 28, 2019

jreback Mar 29, 2019

jreback left a comment

jreback Mar 30, 2019

WillAyd Mar 30, 2019

jreback Mar 30, 2019

jreback commented Mar 30, 2019

		from pandas._libs.tslibs import Period, NaT, Timedelta, Timestamp


		DateTimeLike = Union[datetime, timedelta, Period, Timedelta, Timestamp]


		.. _whatsnew_0250.enhancements.typing:

		Type Hints and ``pandas.typing``

Initial pandas.typing Module #25884

Initial pandas.typing Module #25884

Conversation

WillAyd commented Mar 26, 2019

jreback commented Mar 26, 2019

codecov bot commented Mar 26, 2019

Codecov Report

codecov bot commented Mar 26, 2019

Codecov Report

codecov bot commented Mar 26, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

WillAyd commented Mar 27, 2019 via email

jbrockmendel commented Mar 27, 2019

WillAyd commented Mar 27, 2019 • edited Loading

jbrockmendel commented Mar 27, 2019

WillAyd commented Mar 27, 2019

Choose a reason for hiding this comment

jreback commented Mar 28, 2019

jreback commented Mar 28, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Mar 30, 2019

codecov bot commented Mar 26, 2019 •

edited

Loading

WillAyd commented Mar 27, 2019 •

edited

Loading