'Cannot call function of unknown type' for sequence of callables with different signatures

[waltaskew]

Bug Report

When working with sequences of callables, if all callables in the sequence do not have the same signature mypy will raise false positives when trying to access and call the callables.

To Reproduce

def identity(x: int) -> int:
    return x

def double(x: int) -> int:
    return x * 2

def add(x: int, y: int) -> int:
    return x + y

calls_same_signatures = (
    (identity, (1,)),
    (double, (2,)),
)

calls_different_signatures = (
    (identity, (1,)),
    (add, (1, 2)),
)

for f1, args1 in calls_same_signatures:
    f1(*args1) # type checks

for f2, args2 in calls_different_signatures:
    f2(*args2) # mypy says: Cannot call function of unknown type

Trying to fix this with annotations results in what may be a more revealing error?

from typing import Callable, Sequence, Tuple

def identity(x: int) -> int:
    return x

def double(x: int) -> int:
    return x * 2

def add(x: int, y: int) -> int:
    return x + y

calls_same_signatures: Sequence[Tuple[Callable[..., int], Tuple[int, ...]]]
calls_same_signatures = (
    (identity, (1,)),
    (double, (2,)),
)

calls_different_signatures: Sequence[Tuple[Callable[..., int], Tuple[int, ...]]]
calls_different_signatures = (
    (identity, (1,)),
    (add, (1, 2)),
)

f1: Callable[..., int]
for f1, args1 in calls_same_signatures: # type checks
    f1(*args1)

f2: Callable[..., int]
for f2, args2 in calls_different_signatures:  # mypy says: Incompatible types in assignment (expression has type "function", variable has type "Callable[..., int]")
    f2(*args2)

Expected Behavior

I'd expect this to type check.
I'm not sure if it might be a contravariant vs. covariant thing? That is, does this issue stem from the question over whether the function is a Callable[[int], int] or a Callable[..., int] when it comes out of the sequence?

Actual Behavior

Mypy raises an error when attempting to call functions in calls_different_signatures,
Cannot call function of unknown type in the first example, Incompatible types in assignment (expression has type "function", variable has type "Callable[..., int]") in the second.

Your Environment

• Mypy version used: 0.782
• Mypy command-line flags: none
• Mypy configuration options from mypy.ini (and other config files): none
• Python version used: 3.6.5
• Operating system and version: OS X 10.15.7

[hauntsaninja]

Note, you can get your code to type check by putting the annotation on the same line:

calls_different_signatures: Sequence[Tuple[Callable[..., int], Tuple[int, ...]]] = (
    (identity, (1,)),
    (add, (1, 2)),
)

[waltaskew]

Can also get it to type check by using a List rather than a Sequence

calls_different_signatures: List[Tuple[Callable[..., int], Tuple[int, ...]]]
calls_different_signatures = [
    (identity, (1,)),
    (add, (1, 2)),
]

Which I think does suggest a variance issue? This type checks as well (still using Sequence for the type but defining the data structure with a list rather than a tuple.)

calls_different_signatures: Sequence[Tuple[Callable[..., int], Tuple[int, ...]]]
calls_different_signatures = [
    (identity, (1,)),
    (add, (1, 2)),
]

[hauntsaninja]

I think it's not as much a variance issue, as it is that the invariance of list serendipitously helps you out here.

When you assign to a variable (and the annotation is on a different line [1]), mypy attempts to infer the most specific type possible that is compatible with the annotation. That's why for the following you see such a verbose type on line 18:

calls_different_signatures: Sequence[Tuple[Callable[..., int], Tuple[int, ...]]]
calls_different_signatures = (
    (identity, (1,)),
    (add, (1, 2)),
)
reveal_type(calls_different_signatures)
reveal_type(list(calls_different_signatures))
-----
main.py:18: note: Revealed type is 'Tuple[Tuple[def (x: builtins.int) -> builtins.int, Tuple[builtins.int]], Tuple[def (x: builtins.int, y: builtins.int) -> builtins.int, Tuple[builtins.int, builtins.int]]]'
main.py:19: note: Revealed type is 'builtins.list[Tuple[builtins.function, builtins.tuple[builtins.int]]]'

Now the reveal_type on line 19 (which also applies to your loop). Here mypy is performing what it calls a join, where it tries to describe multiple types as a single type. You see it comes up with builtins.function, not Callable[..., int]. This is the source of your problems, but I'm not sure that it's a bug. It's perilous to infer Any, since that could easily lead to very surprising false negatives (especially since I believe mypy is joining the exact type, which doesn't have any Anys (the ... in a Callable is basically Any)).

Why does it work for list? It's because mypy narrows to the specific type that's compatible with the annotation. That's how variance happily affects you here.

I think the most actionable thing here is mypy doing a better job of listening to your annotation. See [1]

[1] The difference in behaviour when the annotation is on a different line is surprising and has downsides, so we've resolved to change it (see #2008 and a recent discussion on typing-sig). It's still a little unclear what the ideal behaviour is for cases like yours (generics that involve Any), but thanks to your report, we'll take it into account when figuring out what the right tradeoffs are :-)

[waltaskew]

Thanks @hauntsaninja that's a very helpful explanation! Happy to close this if it doesn't seem like a bug.

[derhintze]

A similar phenomenon occurs with dicts instead of Sequences. I'm on Python 3.9.1 and mypy 0.812. Consider the following dict to dispatch on the type of a variable (I don't want to discuss why the dispatch is implemented this way, but has to do with https://bugs.python.org/issue39679):

dispatcher: dict[
    typing.Union[typing.Type[list], typing.Type[str], typing.Type[int]],
    typing.Callable[
        [
            typing.Union[list, int, str],
            str,
        ],
        dict[str, list],
    ],
] = {
    list: list_handling_fun,
    int: int_handling_fun,
    str: str_handling_fun,
}

mypy tells me that

error: Dict entry 0 has incompatible type "Type[List[Any]]": "Callable[[List[Any], str], Dict[str, List[Any]]]"; expected "Union[Type[List[Any]], Type[str], Type[int]]": "Callable[[Union[List[Any], int, str], str], Dict[str, List[Any]]]"

[hauntsaninja]

I think your issue might be different? At least, it looks like list_handling_fun genuinely isn't of the annotated type typing.Callable[[typing.Union[list, int, str], str], dict[str, list]], since it can't take an int or str as the first parameter.

[derhintze]

sorry, turned it upside down in my head. feel free to moderate my comment away :)

[arian-f]

I ran into this or a similar bug by constructing a tuple from typed items like in this gist - could someone check whether this is a duplicate or it's its own thing? TIA!

[hauntsaninja]

It's a duplicate