Instantiation of generic return type in Callable[[...], R] collapse into object instead of proper union types.

[bluenote10]

Bug Report

(I would have assumed that this has already been reported, but after searching for a while I couldn't find an exact match. Apologies if this is a duplicate.)

It looks like if there is a heterogenic generic instantiation of the return type of a Callable[[...], R], the return type R immediately becomes object instead of a union type.

To Reproduce

Example on mypy playground.

from typing import Callable, Sequence, Tuple, TypeVar

R = TypeVar("R")

def f1(*args: Callable[[], R]) -> R:
    return args[0]()

def f2(args: Sequence[Callable[[], R]]) -> R:
    return args[0]()

def f3(args: Tuple[Callable[[], R], ...]) -> R:
    return args[0]()

def f4(a: Callable[[], R], b: Callable[[], R]) -> R:
    return a()
    
def foo() -> int:
    return 42
    
def bar() -> str:
    return "..."
    
x1 = f1(foo, bar)
reveal_type(x1) # expected `int | str` but is `object`

x2 = f2([foo, bar])
reveal_type(x2) # expected `int | str` but is `object`

x3 = f3((foo, bar))
reveal_type(x3) # expected `int | str` but is `object`

x4 = f4(foo, bar)
reveal_type(x4) # expected `int | str` but is `object`

Expected Behavior

I would have hoped that all 4 revealed types become a union type corresponding to the union of the return types of foo and bar, i.e.:

Revealed type is "int | str"

Pyright seems to infer the return type as such.

Actual Behavior

mypy instead converts the return type into object instead of a union.

Revealed type is "builtins.object"

Your Environment

• Mypy version used: 1.5.1
• Mypy command-line flags: none specific
• Mypy configuration options from mypy.ini (and other config files): none specific (mypy playground default)
• Python version used: Tried under 3.11 and 3.8

[bluenote10]

Note that as a work-around it seems possible to explicitly overload up to a certain level:

from typing import Callable, Sequence, Tuple, TypeVar, overload

R = TypeVar("R")
R1 = TypeVar("R1")
R2 = TypeVar("R2")
R3 = TypeVar("R3")

@overload
def f1(__args1: Callable[[], R1]) -> R1:
    ...

@overload
def f1(__args1: Callable[[], R1], __args2: Callable[[], R2]) -> R1 | R2:
    ...

@overload
def f1(__args1: Callable[[], R1], __args2: Callable[[], R2], __args3: Callable[[], R3]) -> R1 | R2 | R3:
    ...

def f1(*args: Callable[[], R]) -> R:
    return args[0]()


def foo() -> int:
    return 42
    
def bar() -> str:
    return "..."
    
reveal_type(f1(foo))
reveal_type(f1(bar))
reveal_type(f1(foo, bar))
reveal_type(f1(foo, bar, lambda: None))

outputs:

main.py:30: note: Revealed type is "builtins.int"
main.py:31: note: Revealed type is "builtins.str"
main.py:32: note: Revealed type is "Union[builtins.int, builtins.str]"
main.py:33: note: Revealed type is "Union[builtins.int, builtins.str, None]"

If the number of varags is expected to be large, this isn't super practical though.

And perhaps the issue here is actually not specific to Callable, but more fundamental. In the following example pyright also seems to be able to infer the union type properly, but mypy says it cannot infer the argument type and falls back to Any:

from typing import TypeVar, Generic

T = TypeVar("T")

class Wrapper(Generic[T]):
    def __init__(self, value: T):
        self.value = value

def f(*args: Wrapper[T]) -> T:
    raise NotImplementedError()
    
reveal_type(f(Wrapper(42), Wrapper("...")))

So is this just a manifestation of #230?

[erictraut]

I think most of this is explained by the fact that mypy uses a "join" operator rather than a union operator. For details, refer to this documentation and this documentation.

There's a tag called topic-join-v-union that marks all of the issues related to this behavior. That same tag should probably be added to this issue.

Your last code sample (with the Wrapper class) can't be explained by a join (since a join will never produce an Any type), so I suspect that's a different bug. @ilevkivskyi has done some excellent work to improve mypy's constraint solver recently, so it's possible this has already been fixed in the master branch.

[bluenote10]

Thanks for pointing that out!

It looks like the central tracking issue for "join v union" is #12056. Not sure if the general policy is to keep individual related issues open, but we might as well close this one in favor of the central one.

[ilevkivskyi]

@bluenote10 The example with Wrapper is actually wrong:

def f(*args: Wrapper[T]) -> T:
    if len(args) == 2:
        first, second = args
        first.value = second.value  # Should be totally OK, since both are `T`, right?
    ... # return whatever

a = Wrapper(1)
b = Wrapper("...")
_ = f(a, b)  # <- mypy gives error here
a.value += 1  # Surprise!

This is why mypy says it can't infer valid value for T, and Any in the result is because there is a general idea to avoid further errors, after we gave one error, thus the most lenient type.

You can actually trick mypy by saying that T is covariant (and mypy will trust you unless you do something egregious like putting T into explicitly contravariant position). This however will get you only as far as inferring object. So yes, this issue is purely about "union-vs-join" thing.

[erictraut]

@ilevkivskyi, I'm not sure what you mean by "Wrapper is actually wrong". If T has the type int | str (or object), I don't see any type violation. Or am I missing something?

a: Wrapper[int | str] = Wrapper(1)
b: Wrapper[int | str] = Wrapper("...")
c1 = f(a, b) # Mypy doesn't emit an error
reveal_type(c1) # Both mypy and pyright reveal "int | str"

c2 = f(Wrapper(1), Wrapper("...")) # Mypy emits an error, pyright doesn't
reveal_type(c2) # Pyright reveals: "int | str"

In the second call to f above, what constraint prevents T from taking on the type int | str? I just want to make sure that pyright isn't doing something incorrect here.