#25790 Updating type hints to Python3 syntax in pandas/core/array #25829

gwrome · 2019-03-22T03:50:01Z

addresses the other half of Update Typing Comments to Python3 Syntax #25790
pandas/tests/arrays/ tests passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff

… core/arrays/

…/array

codecov · 2019-03-22T04:28:27Z

Codecov Report

Merging #25829 into master will decrease coverage by <.01%.
The diff coverage is 97.36%.

@@            Coverage Diff             @@
##           master   #25829      +/-   ##
==========================================
- Coverage   91.27%   91.27%   -0.01%     
==========================================
  Files         173      173              
  Lines       53002    53017      +15     
==========================================
+ Hits        48376    48389      +13     
- Misses       4626     4628       +2

Flag	Coverage Δ
#multiple	`89.83% <97.36%> (ø)`	⬆️
#single	`41.78% <97.36%> (+0.02%)`	⬆️

Impacted Files	Coverage Δ
pandas/core/arrays/base.py	`98.33% <100%> (+0.05%)`	⬆️
pandas/core/arrays/datetimelike.py	`97.69% <100%> (+0.01%)`	⬆️
pandas/core/arrays/integer.py	`96.33% <100%> (+0.01%)`	⬆️
pandas/core/arrays/period.py	`98.53% <100%> (ø)`	⬆️
pandas/core/arrays/sparse.py	`92.19% <100%> (ø)`	⬆️
pandas/core/arrays/datetimes.py	`97.8% <100%> (ø)`	⬆️
pandas/core/arrays/array_.py	`97.87% <75%> (-2.13%)`	⬇️
pandas/util/testing.py	`89.3% <0%> (-0.11%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e31b4f4...ea9c2b3. Read the comment docs.

codecov · 2019-03-22T04:28:28Z

Codecov Report

Merging #25829 into master will increase coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #25829      +/-   ##
==========================================
+ Coverage   91.53%   91.56%   +0.02%     
==========================================
  Files         175      175              
  Lines       52808    52789      -19     
==========================================
- Hits        48338    48335       -3     
+ Misses       4470     4454      -16

Flag	Coverage Δ
#multiple	`90.12% <100%> (+0.02%)`	⬆️
#single	`41.83% <100%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/arrays/datetimes.py	`97.8% <ø> (ø)`	⬆️
pandas/core/arrays/base.py	`98.33% <100%> (+0.05%)`	⬆️
pandas/core/arrays/datetimelike.py	`97.69% <100%> (+0.01%)`	⬆️
pandas/core/arrays/integer.py	`96.33% <100%> (+0.01%)`	⬆️
pandas/core/arrays/period.py	`98.53% <100%> (ø)`	⬆️
pandas/core/arrays/sparse.py	`92.18% <100%> (ø)`	⬆️
pandas/core/arrays/array_.py	`100% <100%> (ø)`	⬆️
pandas/compat/pickle_compat.py	`69.51% <0%> (-6.1%)`	⬇️
pandas/plotting/_compat.py	`83.33% <0%> (-3.34%)`	⬇️
pandas/compat/numpy/__init__.py	`93.1% <0%> (-0.23%)`	⬇️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 882961d...23aba8a. Read the comment docs.

jreback · 2019-03-22T11:30:02Z

pandas/core/arrays/array_.py

-          copy=True,    # type: bool
-          ):
-    # type: (...) -> ExtensionArray
+if TYPE_CHECKING:


why is this needed? conditional imports for type checking are not supportable

As I understand it, we need ExtensionArray to be defined to check this function's return type. Since ExtensionArray is defined elsewhere, we need to import it during type checking. I based this code on the approach described in PEP 484 — Runtime or type checking.

Is there a better, supportable way to do this?

Yea this is the same conversation as #25802 (comment)

@jreback our support of 3.5 includes 3.5.0 and 3.5.1 right? This wasn't introduced until 3.5.2

I would instead simply use the ABCExtensionArray classes which would work for this (and lots of other clases). we should maybe just do this generally. These are by-definition available with no dependencies for this exact purpose.

jreback · 2019-03-23T20:44:31Z

pandas/core/arrays/array_.py

-          copy=True,    # type: bool
-          ):
-    # type: (...) -> ExtensionArray
+if TYPE_CHECKING:


I would instead simply use the ABCExtensionArray classes which would work for this (and lots of other clases). we should maybe just do this generally. These are by-definition available with no dependencies for this exact purpose.

WillAyd · 2019-03-28T02:12:34Z

pandas/core/arrays/base.py

@@ -343,8 +337,7 @@ def astype(self, dtype, copy=True):
        """
        return np.array(self, dtype=dtype, copy=copy)

-    def isna(self):
-        # type: () -> Union[ExtensionArray, np.ndarray]
+    def isna(self) -> Union['ExtensionArray', np.ndarray]:


Might as well use the ABC here and throughout module

WillAyd · 2019-03-28T02:14:55Z

pandas/core/arrays/datetimelike.py

@@ -58,8 +58,7 @@ def _get_attributes_dict(self):
        return {k: getattr(self, k, None) for k in self._attributes}

    @property
-    def _scalar_type(self):
-        # type: () -> Union[type, Tuple[type]]
+    def _scalar_type(self) -> Union[type, Tuple[type]]:


Let's use Type consistently

WillAyd · 2019-03-28T02:15:29Z

pandas/core/arrays/period.py

+            scalars: Sequence[Optional[Period]],
+            dtype: PeriodDtype = None,
+            copy: bool = False,
+    ) -> 'PeriodArray':


ABCPeriodArray

WillAyd · 2019-03-28T02:15:54Z

pandas/core/arrays/sparse.py

-def _sparse_array_op(left, right, op, name):
-    # type: (SparseArray, SparseArray, Callable, str) -> Any
+def _sparse_array_op(
+        left: 'SparseArray',


ABCSparseArray

I think we're good to go @WillAyd @jreback

WillAyd

@gwrome one other thing that could be helpful - since some if not all of these modules are blacklisted from mypy could you manually run locally for the modules and compare this PR to head? Just want to make sure there are no new regressions

pandas/core/arrays/sparse.py

WillAyd

Couple minor things. IMO this is close - obviously some of these annotations are wrong and may need to be fixed but can be done in subsequent PRs

WillAyd · 2019-03-28T05:25:33Z

pandas/core/arrays/period.py

-        # TODO: ABCIndexClass is a valid type for other but had to be excluded
-        # due to length of Py2 compatability comment; add back in once migrated
-        # to Py3 syntax
+            other: Union[ExtensionArray, np.ndarray, ABCIndexClass],


Any reason for removing the type for ndarray?

During my testing at the time (before you added mypy.ini and the rest of the type checking framework to the project), mypy was throwing an error that suggested that ndarrays couldn't take a subscript, so I dropped it. I probably had something in my testing environment set up incorrectly (or maybe more strictly, not sure), because it doesn't throw that error with the current mypy.ini minus the blacklist. Will add it back.

Wait, it wasn't mypy. code_checks doctests throws

...pandas/core/arrays/period.py", line 545, in PeriodArray other: Union[ExtensionArray, np.ndarray[int], ABCIndexClass], TypeError: 'type' object is not subscriptable

Gotcha. Kind of strange this throws an error now but not before. Something to revisit later

pandas/core/arrays/sparse.py

jreback · 2019-03-28T20:23:46Z

pandas/core/arrays/base.py

@@ -343,8 +338,7 @@ def astype(self, dtype, copy=True):
        """
        return np.array(self, dtype=dtype, copy=copy)

-    def isna(self):
-        # type: () -> Union[ExtensionArray, np.ndarray]
+    def isna(self) -> Union[ABCExtensionArray, np.ndarray]:


so this is a the first stab at ArrayLike (e.g. to put into pandas.typing)

Seems like a reasonable place to start.

these are ok here for now, mainly adding things that we should add

Agreed this would be great @gwrome do you mind opening up an issue as a follow up?

No problem. I should be able to do that this weekend.

jreback · 2019-03-28T20:41:09Z

pandas/core/arrays/sparse.py

-        # type: (Union[str, np.dtype, 'ExtensionDtype', type], Any) -> None
+    def __init__(
+            self,
+            dtype: Union[str, np.dtype, ExtensionDtype, Type] = np.float64,


this could be Dtype (in pandas.typing)

jreback · 2019-03-30T18:35:25Z

lgtm. @WillAyd merge when ready

WillAyd · 2019-03-30T19:21:33Z

nice job @gwrome - thanks for helping to kick off these typing initiatives

gwrome added 9 commits March 21, 2019 22:31

pandas-dev#25790 Updated type hints to Python3 style in files outside…

a0b8c4c

… core/arrays/

pandas-dev#25790 Added TypeVar to hint in core/indexes/base.py

46cc909

Fixed reference to types.Tuple that should have been to typing.Tuple

7acb89a

Fixed import sort

d9c53ab

pandas-dev#25802 Updated type hints to python3 syntax (array files)

e73fe18

Removed instance variable type hint for 3.5 compatibility

545d38d

Cleaned up array/ type hints

abc09fa

pandas-dev#25790 Updating type hints to Python3 syntax in pandas/core…

c37ef88

…/array

pandas-dev#25790 Updating type hints to Python3 syntax in pandas/core…

ea9c2b3

…/array

jreback requested changes Mar 22, 2019

View reviewed changes

jreback requested changes Mar 23, 2019

View reviewed changes

Remove conditional import

20b63e4

gfyoung added the Typing type annotations, mypy/pyright type checking label Mar 25, 2019

WillAyd requested changes Mar 28, 2019

View reviewed changes

gwrome added 4 commits March 27, 2019 22:57

Change forward refs to ABCs

cf774d7

Merge branch 'master' into update-array-type-hints-python3

50db810

Merge branch 'master' into array-type-hints-python3

92490a8

Change forward refs to ABCs

441f4bd

WillAyd reviewed Mar 28, 2019

View reviewed changes

pandas/core/arrays/sparse.py Show resolved Hide resolved

Change remaining forward refs to ABCs

33db1ee

WillAyd requested changes Mar 28, 2019

View reviewed changes

type to Type

23aba8a

jreback reviewed Mar 28, 2019

View reviewed changes

jreback added this to the 0.25.0 milestone Mar 28, 2019

jreback mentioned this pull request Mar 28, 2019

Initial pandas.typing Module #25884

Merged

jreback approved these changes Mar 30, 2019

View reviewed changes

WillAyd approved these changes Mar 30, 2019

View reviewed changes

WillAyd merged commit f0ba498 into pandas-dev:master Mar 30, 2019

gwrome deleted the array-type-hints-python3 branch March 31, 2019 14:46

gwrome mentioned this pull request Mar 31, 2019

Add ArrayLike an Dtype definitions to pandas._typing #25942

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#25790 Updating type hints to Python3 syntax in pandas/core/array #25829

#25790 Updating type hints to Python3 syntax in pandas/core/array #25829

gwrome commented Mar 22, 2019

codecov bot commented Mar 22, 2019

codecov bot commented Mar 22, 2019 •

edited

Loading

jreback Mar 22, 2019

gwrome Mar 22, 2019

WillAyd Mar 22, 2019

jreback Mar 23, 2019

jreback Mar 23, 2019

WillAyd Mar 28, 2019

WillAyd Mar 28, 2019

WillAyd Mar 28, 2019

WillAyd Mar 28, 2019

gwrome Mar 28, 2019

WillAyd left a comment

WillAyd left a comment

WillAyd Mar 28, 2019

gwrome Mar 28, 2019

gwrome Mar 28, 2019

WillAyd Mar 30, 2019

jreback Mar 28, 2019

gwrome Mar 29, 2019

jreback Mar 30, 2019

WillAyd Mar 30, 2019

gwrome Mar 30, 2019

jreback Mar 28, 2019

jreback commented Mar 30, 2019

WillAyd commented Mar 30, 2019

#25790 Updating type hints to Python3 syntax in pandas/core/array #25829

#25790 Updating type hints to Python3 syntax in pandas/core/array #25829

Conversation

gwrome commented Mar 22, 2019

codecov bot commented Mar 22, 2019

Codecov Report

codecov bot commented Mar 22, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd left a comment

Choose a reason for hiding this comment

WillAyd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Mar 30, 2019

WillAyd commented Mar 30, 2019

codecov bot commented Mar 22, 2019 •

edited

Loading