Allow @overload outside stubs

[bdarnell]

Mypy does not consider the stub file for the module it is currently type-checking (and doing so was apparently considered and rejected recently). However, @overload as defined in the PEP is only allowed to appear in stubs, which implies a requirement for type checkers to consider a module's own stubs when type checking it. Otherwise, the ability to type check a module which defines an overloaded function is limited.

My motivating example:

@overload
def utf8(s: None) -> None: ...
@overload
def utf8(s: AnyStr) -> bytes: ...

This could be expressed as def utf8(s: Optional[AnyStr]) -> Optional[bytes], but I'm trying to capture the fact that if the argument is not None then the result is also guaranteed not to be None, so it doesn't introduce unnecessary Optional-ness. (Is there a way to express this with a TypeVar? I don't see one). I can put the overloads in a stub for the benefit of external callers of this module, but then I need to put all the type information for this module in the stub and can't type check it internally (I could duplicate all the non-overload type info in the module itself and get partial coverage). I could also put overloaded functions in their own tiny modules to minimize the impact, but that just feels silly.

[gvanrossum]

I think it can be done with typevars: AnyStr is defined as TypeVar('AnyStr', str, bytes). It's not special in any way though. You should be able to define your own typevar as OptAnyStr = TypeVar('OptAnyStr', None, str, bytes). I think we're not in principle against eventually considering the .pyi and .py files together, but it would be a lot of work to add this to mypy, and your use case (@overload) is pretty much the only one. (I vaguely recall another issue discussing this, but I can't find it here. Maybe it's in the typeshed repo or even in typehinting, which is for PEP 484 and the PyPI version of typing.py.)

[JukkaL]

I'm sure there's a better way to support @overload than having a separate .pyi file (such as the approach discussed in python/typing#72). Separate .pyi files will be painful to maintain, and they don't provide most of the readability gains of inline annotations.

[bdarnell]

I don't see how OptAnyStr would help. Note that there are two different unions involved: (Union[bytes, unicode, None]) -> Union[bytes, None]. I want to constrain the return type as a function of the input type, but it can't just pick the same type on both sides because of the (unicode) -> bytes case.

I'd be happy with a non-stub way to express overloads (and would prefer it to a requirement for separate stub files to use this feature). The # type: comment syntax could support this relatively easily and wouldn't carry the implications of runtime dispatch like a decorator would.

Overloads that can't be satisfied with TypeVars may be rare, but when they come up they can be significant (that's why the feature is in the PEP). In this case, tornado.escape.utf8 is used all over Tornado, and I'd have to add dozens of explicit casts if I had to use the union declaration instead of the overloaded version.

The interaction between stub files and source files is generally awkward. I just realized that the type annotations I've been adding to Tornado source files aren't actually used outside those files because there are stubs for Tornado in typeshed (mypy --verbose should report where it's loading type information from). Assuming I release a version of Tornado with inline type annotations I'm going to need to be able to make sure that those types take precedence over the versions in typeshed.

[gvanrossum]

Wow, you're really bleeding edge here. Great to see you exploring this and helping us find the gray areas. Short term, the only solution I can think of for utf8() would be to move it to a separate file and have a stub for that file containing the @overload declarations. That's awkward, but it can work with today's mypy. Longer term, this example suggests that we really need to thing harder about allowing @overload in non-stub files. That's a PEP 484 change and we should discuss it on python-dev, but we might as well first prepare a proposal. I've started laying out the problem in python/typing#175 -- let's continue the discussion there. Regarding inline annotations vs typeshed, that's another unsolved problem. I've opened python/typeshed#52 for it, but I frankly don't know how to solve it. Please join those discussions!

[gvanrossum]

Actually, as I wrote in python/typeshed#52, the stubs vs. real code problem is easily solved by setting MYPYPATH to include the version of Tornado you're using -- the stubs in typeshed are searched last. The solution for @overload utf8() is to change the PEP, which I will do shortly.

[gvanrossum]

The PEP is now updated. This issue should stay open until it's been implemented in mypy.

[rwbarton]

What's the idea for how this would actually work in a user program? After all @overload doesn't (and can't I think, especially with Python 2-style type comments) do runtime type-based dispatch.

I think there needs to be just one function body, that mypy will type check with each of the overloaded types. Straw-man syntax in a minimal example:

@overload
def f(x: str) -> str: ...    # literal ...
@overload
def f(x: int) -> int:
    return x

At runtime @overload would be just the identity function and the second definition of f would overwrite the first one. (Maybe for Python 3 @overload could collect the type attributes and store them in a property of the returned function.)

[rwbarton]

Oh interesting, there's actually an example of the expected (but not yet implemented) syntax in the docstring for @overload, which is a bit different.

      @overload
      def utf8(value: None) -> None: ...
      @overload
      def utf8(value: bytes) -> bytes: ...
      @overload
      def utf8(value: str) -> bytes: ...
      def utf8(value):
          # implementation goes here

[gvanrossum]

Yeah, that's how it should work. The overloads are for the type checker, the final non-overloaded function is for the interpreter. If the latter has annotations it should be checked as a regular function; separately mypy might want to check that it can handle all the overloads; when type-checking a call, only the overloads should be considered.

[cpennington]

I was looking into implementing this. From what I can see, the example above (utf8) don't typecheck in mypy as of v0.4.5, with the following errors:

$ mypy test_overloads.py
test_overloads.py:4: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
test_overloads.py:4: error: Overloaded function signatures 1 and 3 overlap with incompatible return types
test_overloads.py:9: error: Name 'utf8' already defined

The error on line 9 can be fixed by adding an @overload to the final definition of utf8, but it's not clear to me if that's allowed by the spec or not.

The errors about overlapping signatures are because None inhabits all types, unless --strict-optional is used (and with --strict-optional, the errors go away).

Is there a reason for not putting the implementation in the final @overloaded definition? Also, without --strict-optional, should we ever expect an overload involving None to typecheck?

[gvanrossum]

I guess the reason for the "overlap" errors is that that example can only work with --strict-optional. If the final overload also had the implementation, it would be hard to type-check its body. Also we want to syntactically distinguish between stubs (where there is no implementation) and non-stubs (where there must be an implementation).

[gvanrossum]

@sixolet Adding you to this issue for coordination purposes.

[douglas-treadwell]

I don't understand how this solved #2305.

I'm using MyPy 0.511 and still having the issue described in #2305.

[ilevkivskyi]

It doesn't. Guido was referring to python/typeshed#1136 (on typeshed tracker)

[douglas-treadwell]

Although 0.520-dev (GitHub master) does seem to handle #2305...

Maybe that's because installing MyPy from GitHub master requires preparing the latest typeshed?

[douglas-treadwell]

@ilevkivskyi can you clarify?

[ilevkivskyi]

mypy master is synced with typeshed every two-three days. PR python/typeshed#1136 was merged more than two months ago.