Make overloads support classmethod and staticmethod #5224

Michael0x2a · 2018-06-15T20:51:11Z

This pull request adds support for mixing classmethods, staticmethods, and overloads.

It does so by adding in some extra logic into the semantic analysis phase to detect when an overload is using classmethods or staticmethods and adds that information into OverloadedFuncDef. This allowed me to add is_class and is_static fields to FuncBase, the common base class for overloads and other function-related things.

This PR does not attempt to handle overloads + other arbitrary decorators.

Resolves #328 (which unblocks #2254?)

This commit moves the `is_class` and `is_static` fields into FuncBase. It also cleans up the list of flags so they don't repeat the 'is_property' entry, which is now present in `FUNCBASE_FLAGS`. The high-level plan is to modify the `is_class` and `is_static` fields in OverloadedFuncDef for use later in mypy.

This commit adjusts the semantic analysis phase to detect and record when an overload appears to be a classmethod or staticmethod.

This commit modifies mypy to use the `is_static` and `is_class` fields of OverloadedFuncDef as appropriate. I found the code snippets to modify by asking PyCharm for all instances of code using those two fields and modified the surrounding code as appropriate.

Both the attrs and dataclasses plugins manually patch classmethods -- we do the same for overloads.

ilevkivskyi

Thanks! Here are few comments.

ilevkivskyi · 2018-06-15T23:07:05Z

mypy/nodes.py

@@ -481,9 +490,9 @@ def set_line(self, target: Union[Context, int], column: Optional[int] = None) ->
        self.variable.set_line(self.line, self.column)


-FUNCITEM_FLAGS = [
+FUNCITEM_FLAGS = FUNCBASE_FLAGS + [


This change adds is_property, is this intentional?

It is. I made this change partly on principle since is_property actually is a field of FuncBase -- it felt cleaner to just force all subclasses to preserve that field no matter what.

This change also doesn't actually change the serialized output in practice. FuncItem currently has only two subtypes: FuncDef and LambdaExpr. The former subclass previously explicitly set and serialized the is_property field so this change makes no difference there. The latter subclass never really uses is_property but also doesn't have any serialize/deserialize methods, which makes this change moot.

ilevkivskyi · 2018-06-15T23:10:15Z

mypy/plugins/attrs.py

+                if stmt.impl is not None:
+                    assert isinstance(stmt.impl, Decorator)
+                    if isinstance(stmt.impl.func.type, CallableType):
+                        stmt.impl.func.type.arg_types[0] = class_type


It's good that you also take care about plugins.

ilevkivskyi · 2018-06-15T23:12:39Z

mypy/nodes.py

@@ -370,13 +370,20 @@ def __str__(self) -> str:
        return 'ImportedName(%s)' % self.target_fullname


+FUNCBASE_FLAGS = [
+    'is_property', 'is_class', 'is_static',
+]


Shouldn't we also update astdiff.py according to these flag reshuffling? This may break fine grained incremental mode (in some corner cases).

Ooh, good point -- I didn't even know that file existed.

I tried making the change + tried adding an test to one of the fine-grained incremental tests. (I'm pretty unfamiliar with fine-grained incremental stuff though, so let me know if I did it incorrectly.)

ilevkivskyi · 2018-06-15T23:14:36Z

mypy/semanal.py

+            elif isinstance(item, FuncDef):
+                inner = item
+            else:
+                raise AssertionError()


Please add a message to all assertion errors. Typically we write assert False, "Impossible blah-blah-blah".

ilevkivskyi · 2018-06-15T23:16:29Z

test-data/unit/check-attr.test

+@attr.s
+class A:
+    a: int
+    b: str


I would rather make these attr.ibs, to check that the cls signature below is generated correctly by the plugin.

ilevkivskyi · 2018-06-15T23:25:51Z

test-data/unit/check-overloading.test

+    def foo(cls, x): pass
+
+class BadChild(Parent):
+    @overload  # E: Signature of "foo" incompatible with supertype "Parent"


Maybe add few more tests to check overriding an overloaded instance method, with overloaded class methods and vice versa?

Ah whoops, I completely forgot to handle this case. Fixed.

ilevkivskyi · 2018-06-15T23:28:23Z

mypy/checker.py

@@ -1290,8 +1290,8 @@ def check_override(self, override: FunctionLike, original: FunctionLike,
            fail = True

        if isinstance(original, CallableType) and isinstance(override, CallableType):
-            if (isinstance(original.definition, FuncItem) and
-                    isinstance(override.definition, FuncItem)):
+            if (isinstance(original.definition, FuncBase) and


Note that this PR (in particular because of this change) may have conflicts with my type aliases refactoring. In turned out that the type_override (that I killed) was also used for some overload hacks.

I'm not sure I'm completely following -- is there now going to be some major difference between FuncItem and FuncBase?

(I also ended up streamlining this code to handle the "overriding a classmethod with an instance method" case more cleanly -- not sure if that change makes the potential conflict better or worse.)

This commit: 1. Updates astdiff.py and adds a case to one of the fine-grained dependency test files. 2. Adds some helper methods to FunctionLike. 3. Performs a few misc cleanups.

ilevkivskyi

Thanks! Here are few more comments. I just checked locally your changes work well with my refactoring.

ilevkivskyi · 2018-06-16T15:47:34Z

mypy/checker.py

        return func.is_static
-    return False
+    raise AssertionError("Unexpected func type: {}".format(type(func)))


Could you please reformat this as assert False, "message"?

ilevkivskyi · 2018-06-16T16:41:27Z

mypy/nodes.py

 class FuncBase(Node):
    """Abstract base class for function-like nodes"""

    __slots__ = ('type',
                 'unanalyzed_type',
                 'info',
                 'is_property',
+                 'is_class',


Could you please add a comment # Uses @classmethod? as before (and probably also for is_static) below. Currently we have is_class, is_classmethod, and is_classmethod_class (the latter I would say is quite bad name), and I want to avoid potential confusions.

ilevkivskyi · 2018-06-16T16:42:35Z

mypy/semanal.py

+            elif isinstance(defn.impl, FuncDef):
+                inner = defn.impl
+            else:
+                raise AssertionError()


Could you also please add an assertion message here, ad reformat like above via assert False?

ilevkivskyi · 2018-06-16T16:51:41Z

mypy/server/astdiff.py

@@ -173,7 +173,7 @@ def snapshot_definition(node: Optional[SymbolNode],
            signature = snapshot_type(node.type)
        else:
            signature = snapshot_untyped_signature(node)
-        return ('Func', common, node.is_property, signature)
+        return ('Func', common, node.is_property, node.is_class, node.is_property, signature)


is_property now appears twice in the list, did you mean is_static? Also maybe the isinstance above can use FuncBase?

ilevkivskyi · 2018-06-16T17:06:16Z

test-data/unit/check-overloading.test

+    @classmethod
+    def foo(cls, x: int) -> int: ...
+    @overload
+    @classmethod


One more suggestion: could you please add to your TODO list adding few tests to check that overloads work well with self-types (both instance, and class methods using cls: Type[T])?

ilevkivskyi · 2018-06-16T17:11:07Z

test-data/unit/fine-grained.test

+main:3: error: Revealed type is 'builtins.int'
+==
+main:3: error: Revealed type is 'Any'
+main:3: error: No overload variant of "foo" of "Wrapper" matches argument type "int"


Yes, this test looks good.

Michael0x2a · 2018-06-16T18:41:04Z

@ilevkivskyi -- ok, done. I went ahead and just added some self-type tests to this commit.

Michael0x2a added 4 commits June 15, 2018 13:21

Make semantic analysis phase record class/static methods with overloads

4947e9f

This commit adjusts the semantic analysis phase to detect and record when an overload appears to be a classmethod or staticmethod.

Add support for overloaded classmethods in attrs/dataclasses

81dfa69

Both the attrs and dataclasses plugins manually patch classmethods -- we do the same for overloads.

ilevkivskyi reviewed Jun 15, 2018

View reviewed changes

ilevkivskyi self-assigned this Jun 15, 2018

Respond to code review

bfaba34

This commit: 1. Updates astdiff.py and adds a case to one of the fine-grained dependency test files. 2. Adds some helper methods to FunctionLike. 3. Performs a few misc cleanups.

ilevkivskyi reviewed Jun 16, 2018

View reviewed changes

Respond to code review; add tests for self types

563748a

ilevkivskyi approved these changes Jun 16, 2018

View reviewed changes

ilevkivskyi merged commit 29889c8 into python:master Jun 16, 2018

ilevkivskyi mentioned this pull request Jun 21, 2018

Regression in classmethod override #5259

Closed

Michael0x2a deleted the make-overloads-support-classmethod-and-staticmethod branch July 9, 2018 04:18

nautics889 mentioned this pull request Oct 23, 2019

Overloading staticmethods #7781

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make overloads support classmethod and staticmethod #5224

Make overloads support classmethod and staticmethod #5224

Michael0x2a commented Jun 15, 2018

ilevkivskyi left a comment

ilevkivskyi Jun 15, 2018

Michael0x2a Jun 16, 2018

ilevkivskyi Jun 15, 2018

ilevkivskyi Jun 15, 2018

Michael0x2a Jun 16, 2018

ilevkivskyi Jun 15, 2018

Michael0x2a Jun 16, 2018

ilevkivskyi Jun 15, 2018

Michael0x2a Jun 16, 2018

ilevkivskyi Jun 15, 2018

Michael0x2a Jun 16, 2018

ilevkivskyi Jun 15, 2018

Michael0x2a Jun 16, 2018

ilevkivskyi left a comment

ilevkivskyi Jun 16, 2018

ilevkivskyi Jun 16, 2018

ilevkivskyi Jun 16, 2018 •

edited

Loading

ilevkivskyi Jun 16, 2018

ilevkivskyi Jun 16, 2018

ilevkivskyi Jun 16, 2018

Michael0x2a commented Jun 16, 2018

Make overloads support classmethod and staticmethod #5224

Make overloads support classmethod and staticmethod #5224

Conversation

Michael0x2a commented Jun 15, 2018

ilevkivskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi Jun 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michael0x2a commented Jun 16, 2018

ilevkivskyi Jun 16, 2018 •

edited

Loading