Decorator factory function typing

[ssamojl]

Bug Report

I have a decorator factory function that takes a schema and returns a decorator that validates something against the provided schema and also passes in an argument to the wrapped function. Here's a simplified version of the code:

from typing import Any, Callable, Concatenate, ParamSpec, TypeVar

P = ParamSpec("P")
Schema = TypeVar("Schema") 
Ret = TypeVar("Ret") 

def validate(
    schema: Schema,
) -> Callable[
    [Callable[Concatenate[int, P], Ret]],
    Callable[P, Ret],
]:
    def decorator(
        f: Callable[
            Concatenate[int, P], Ret
        ]
    ) -> Callable[P, Ret]:
        def wrap(*args: P.args, **kwargs: P.kwargs) -> Ret: 
            # for simplicity, omitting schema validation 
            # that would normally happen here.
            
            additional_arg = 42 # also simplified
            return f(additional_arg, *args, **kwargs)

        return wrap

    return decorator

With mypy 1.1.1, the type annotations are fine. But with the recently released mypy 1.2.0, I get this error:

error: Incompatible return value type (got "Callable[[Arg(Callable[[int, **P], Ret], 'f')], Callable[P, Ret]]", expected "Callable[[Callable[[int, **P], Ret]], Callable[P, Ret]]")  [return-value]

To Reproduce

Run the following with mypy 1.2.0 (example returns an error) and mypy 1.1.1 (no error):

• Playground URL: https://mypy-play.net/?mypy=1.2.0&python=3.11&gist=c8cc63e9ae1cc5b82fc0b613a26aa302
• Gist URL: https://gist.github.com/mypy-play/c8cc63e9ae1cc5b82fc0b613a26aa302

Expected Behavior

I'd expect mypy to not return an error unless there is a legitimate issue with the types in this code (if that's the case, please let me know!).

Moreover, the actual type in the error message doesn't seem to be a valid type (specifically Arg(Callable[[int, **P], Ret], 'f') in this example).

Actual Behavior

error: Incompatible return value type (got "Callable[[Arg(Callable[[int, **P], Ret], 'f')], Callable[P, Ret]]", expected "Callable[[Callable[[int, **P], Ret]], Callable[P, Ret]]") [return-value]

Your Environment

• Mypy version used: 1.2.0
• Mypy command-line flags: none
• Mypy configuration options from mypy.ini (and other config files): n/a
• Python version used: 3.11

[dgoldstein0]

suggestion: I've had better luck with Protocol where I give a __call__ signature, than with Callable. doesn't necessarily address your issue but may give you a way to write the code that works better with mypy.

[hauntsaninja]

Thanks for the report!

From a glance, it seems like the type variable scoping isn't aligned with what you're expecting. E.g. in the following version you get the fun error Incompatible return value type (got "Callable[[Callable[[int, **P], Ret]], Callable[P, Ret]]", expected "Callable[[Callable[[int, **P], Ret]], Callable[P, Ret]]"):

from typing import Any, Callable, Concatenate, ParamSpec, TypeVar

P = ParamSpec("P")
Ret = TypeVar("Ret") 

def validate() -> Callable[
    [Callable[Concatenate[int, P], Ret]],
    Callable[P, Ret],
]:
    def decorator(f: Callable[Concatenate[int, P], Ret], /) -> Callable[P, Ret]:
        def wrap(*args: P.args, **kwargs: P.kwargs) -> Ret: 
            return f(42, *args, **kwargs)
        return wrap

    return decorator

The difference between 1.1.1 and 1.2 bisects to #14677 , but it's also very possible that 1.1.1 wasn't handling things correctly, didn't look to see.

cc @A5rocks the paramspec fixer, in case they have thoughts!

[gandhis1]

I'm seeing the same issue, a new (false?) negative in Mypy 1.20 that was not present in 1.1:

error: Incompatible return value type (got "Callable[[Arg(Callable[[Table, **P], Executable], 'func')], Callable[P, Result]]", expected "Callable[[Callable[[Table, **P], Executable]], Callable[P, Result]]") [return-value]

I've replaced some of the unneeded parts here with "..." for brevity:

def second_order_decorator(...) -> Callable[[Callable[Concatenate[sa.Table, P], sa.sql.Executable]], Callable[P, sa.engine.Result]]:
    def first_order_decorator(func: Callable[Concatenate[sa.Table, P], sa.sql.Executable]) -> Callable[P, sa.engine.Result]:
        @wraps(func)
        def wrapped_func(*args: P.args, **kwargs: P.kwargs) -> sa.engine.Result:
            return ...   # for brevity

        return wrapped_func

    return first_order_decorator

[A5rocks]

I was preparing to have to dig into the codebase to figure out what went wrong but I just realized what might have happened: I may have somehow leaked strict-mode concatenate handling into non-strict-mode.

While typing that up it turns out my proposed solution actually doesn't fix it. Heh. IIRC this issue is what I mentioned in #14677 (comment) which gets fixed by my followup #14903 (at least the error gets removed) which I really need to get around to finalizing! In fact now that I check, this specific error pops up in the test case I added when checked against current mypy!

[abid-mujtaba]

Looks like the fix was reverted: #15272 (comment). Can we please re-open this issue so that we can keep track of the problem? Thanks.

[A5rocks]

FYI, #15837 also fixes this, so IDK if you want to reopen just for tracking that or what.

[abid-mujtaba]

Subscribing to the PR should suffice, thanks.