Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

overload with restrictions on type arguemnts of self #5320

Closed
wabu opened this issue Jul 4, 2018 · 2 comments · Fixed by #7860
Closed

overload with restrictions on type arguemnts of self #5320

wabu opened this issue Jul 4, 2018 · 2 comments · Fixed by #7860

Comments

@wabu
Copy link

wabu commented Jul 4, 2018

I'd like to be able for overload to handle restrictions on self to the type arguments of a Generic type, but I think mypy ignores restriction on self. Here's an example, where I want to have a different signature for the Lnk.__rshift__ method when the second type argument is a Tuple:

In = TypeVar('In')
Out = TypeVar('Out')
Other = TypeVar('Other')

_1 = TypeVar('_1')
_2 = TypeVar('_2')

__1 = TypeVar('__1')
__2 = TypeVar('__2')

class Lnk(Generic[In, Out]):
    # normal restriction on second argument
    @overload
    def __rshift__(self, other: 'Lnk[Out, Other]') -> 'Lnk[In,Other]':
        pass

    # restriction of type argument of self, as I want to have a special case for links to tuples
    @overload
    def __rshift__(self: 'Lnk[In, Tuple[_1, _2]]',
                   other: 'Tuple[Lnk[_1, __1], Lnk[_2, __2]]'
                  ) -> 'Lnk[In, Tuple[__1, __2]]':
        pass

    def __rshift__(self: Any, other: Any) -> Any:
        ...

# create some links
a: Lnk[str, Tuple[str, int]] = Lnk()
b: Lnk[str, int] = Lnk()
c: Lnk[int, float] = Lnk()

# valid calls to link things together
d: Lnk[str, float] = b >> c
e: Lnk[str, Tuple[int, float]] = a >> (b, c)

# invalid call, but mypy does not check type of self
f: Lnk[str, Tuple[float, int]] = a >> (c, b)

If you create a top level function link with the exact signatures as __rshift__, mypy correctly complains about the invalid call, but I'd like to use the operators to create a DSL:

@overload
def link(self: 'Lnk[In, Out]', other: 'Lnk[Out, Other]') -> 'Lnk[In,Other]':
        pass

@overload
def link(self: 'Lnk[In, Tuple[_1, _2]]',
               other: 'Tuple[Lnk[_1, __1], Lnk[_2, __2]]'
              ) -> 'Lnk[In, Tuple[__1, __2]]':
    pass

def link(self: Any, other: Any) -> Any:
    ...

# mypy correctly complains
f: Lnk[str, Tuple[float, int]] = link(a, (c, b))
  • Python 3.6.5
  • mypy 0.620+dev-f5058bcb269a55b33a02c3e3fd345541839bf06e
@Michael0x2a
Copy link
Collaborator

Here's a more simplified repo -- I think this is ultimately an issue having to do with typevars and self, and not necessarily overloads:

from typing import *

In = TypeVar('In')
Out = TypeVar('Out')

Mid = TypeVar('Mid')
NewOut = TypeVar('NewOut')

class Lnk(Generic[In, Out]):
    def test(self: 'Lnk[In, Mid]', other: 'Lnk[Mid, NewOut]') -> 'Lnk[In, NewOut]':
        pass

class X: pass
class Y: pass
class Z: pass

a: Lnk[X, Y] = Lnk()
b: Lnk[Y, Z] = Lnk()

# Correctly typechecks
a.test(b)

# Currently accepted by mypy, should be an error
b.test(a)

I suspect this is probably a duplicate of #2354? I could be mistaken -- I'm not super familiar with that corner of the codebase.

@wabu
Copy link
Author

wabu commented Jul 4, 2018

Perhaps with an implementation of #2354, e.g. something like a higher order self type variable, one can use it to create a self type TypeVar('SelfT', bound='Lnk') and use that to restrict the arguments to the self type:

@overload
def __rshift__(self: 'SelfT[In, Tuple[_1, _2]]',
               other: 'Tuple[Lnk[_1, __1], Lnk[_2, __2]]') -> 'Lnk[In, Tuple[__1, __2]]':
    ...

But I'd guess the logic of overload also has to be adapted to know about the concept of higher order self types.

Moreover it might be unnecessary to introduce a new type variable for the self-type in this example, as I don't want a specific sub-type of Lnk as a second argument or to be returned, so the SelfT type variable is never 'referenced', only 'bound' by the annotation on self.

ilevkivskyi added a commit to ilevkivskyi/mypy that referenced this issue Nov 5, 2019
Fixes python#3625
Fixes python#5305
Fixes python#5320
Fixes python#5868
Fixes python#7191
Fixes python#7778
Fixes python/typing#680

So, lately I was noticing many issues that would be fixed by (partially) moving the check for self-type from definition site to call site.

This morning I found that we actually have such function `check_self_arg()` that is applied at call site, but it is almost not used. After more reading of the code I found that all the patterns for self-types that I wanted to support should either already work, or work with minimal modifications. Finally, I discovered that the root cause of many of the problems is the fact that `bind_self()` uses wrong direction for type inference! All these years it expected actual argument type to be _supertype_ of the formal one.

After fixing this bug, it turned out it was easy to support following patterns for explicit self-types:
* Structured match on generic self-types
* Restricted methods in generic classes (methods that one is allowed to call only for some values or type arguments)
* Methods overloaded on self-type
* (Important case of the above) overloaded `__init__` for generic classes
* Mixin classes (using protocols)
* Private class-level decorators (a bit hacky)
* Precise types for alternative constructors (mostly already worked)

This PR cuts few corners, but it is ready for review (I left some TODOs). Note I also add some docs, I am not sure this is really needed, but probably good to have.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants