(regression) Interaction between __new__ and Generics

[finite-state-machine]

Bug Report

Consider the following code [mypy-play.net]:

from __future__ import annotations

from typing import *


T = TypeVar('T')


class Parent(Sequence[T]):
    '''a sequence type which is implemented one of two ways,
    according to the input parameters
    '''
    def __new__(cls, model: Sequence[T]) -> Parent[T]:
        if cls is Parent:
            if len(model) % 2:
                return Odds(model)
            return Evens(model)
        return super().__new__(cls)

    _data: Sequence[T]

    @overload
    def __getitem__(self, index: int, /) -> T: ...
    @overload
    def __getitem__(self, index: slice, /) -> Sequence[T]: ...

    def __getitem__(self, index: Union[int, slice], /
            ) -> Union[Sequence[T], T]:  # pragma: no cover

        return self._data[index]

    def __len__(self) -> int:
        return len(self._data)


class Evens(Parent[T]):
    '''the implementation of 'Parent' used when the input has an even length
    '''
    def __new__(cls, model: Sequence[T]) -> Evens[T]:
        ret = super().__new__(cls, model)
                # mypy: error: Argument 1 to "__new__" of "Parent" has
                #              incompatible type "Type[Evens[T]]";
                #              expected "Type[Parent[T]]"
                #              [arg-type]
                #       error: Argument 2 to "__new__" of "Parent" has
                #              incompatible type "Sequence[T]"; expected
                #              "Sequence[T]"
                #              [arg-type]
        assert isinstance(ret, Evens)
        return ret

    def __init__(self, model: Sequence[T]):
        self._data = model[::2]

class Odds(Parent[T]):
    '''the implementation of 'Parent' used when the input has an odd length
    '''
    def __new__(cls, model: Sequence[T]) -> Odds[T]:
            # mypy: two (3) errors, analogous to those of
            #       'Evens.__new__()', above
        ret = super().__new__(cls, model)
        assert isinstance(ret, Odds)
        return ret

    def __init__(self, model: Sequence[T]):
        self._data = model[1::2]


def test_function() -> None:
    # this test passes; the code does work!

    evens = Parent(list(range(10)))
    assert isinstance(evens, Parent)
    assert isinstance(evens, Evens)
    assert list(evens) == [0, 2, 4, 6, 8]

    odds = Parent(list(range(11)))
    assert isinstance(odds, Parent)
    assert isinstance(odds, Odds)
    assert list(odds) == [1, 3, 5, 7, 9]

To Reproduce

See above.

Actual Behavior

main.py:40: error: Argument 1 to "__new__" of "Parent" has incompatible type "Type[Evens[T]]"; expected "Type[Parent[T]]"  [arg-type]
main.py:40: error: Argument 2 to "__new__" of "Parent" has incompatible type "Sequence[T]"; expected "Sequence[T]"  [arg-type]
main.py:61: error: Argument 1 to "__new__" of "Parent" has incompatible type "Type[Odds[T]]"; expected "Type[Parent[T]]"  [arg-type]
main.py:61: error: Argument 2 to "__new__" of "Parent" has incompatible type "Sequence[T]"; expected "Sequence[T]"  [arg-type]
Found 4 errors in 1 file (checked 1 source file)

Expected Behavior

There should be no errors. (In fact, this worked correctly until mypy 0.950.)

The first and third errors make little sense since Type[Evens[T]] is a special case of Type[Parent[T]].

The second and fourth errors make even less sense, since Sequence[T] is exactly the same as Sequence[T].

If there is actually some kind of problem here, the error messages need to be much more helpful than they currently are.

Your Environment

• Mypy version used: master, 0.981
• Mypy command-line flags: --strict --warn-unreachable --show-error-codes
• Mypy configuration options from mypy.ini (and other config files): N/A
• Python version used: 3.10, 3.8

[ilevkivskyi]

Note that this is specific to __new__, other (explicit) class methods work correctly. I didn't check, but it may be caused by #12590 cc @JukkaL FWIW PR looks correct, probably it just exposes some old missing special casing for __new__. Likely should be an easy fix.

[finite-state-machine]

FWIW I've confirmed this issue persists on the development version of mypy (2023-06-26).

[finite-state-machine]

It looks like @kourbou fixed this with 1dd8e7f (#16670) when they fixed #16668:

commit 1dd8e7fe6
Author: Kouroche Bouchiat <[email protected]>
Date:   Sun Dec 17 21:32:57 2023 +0100

    Substitute type variables in return type of static methods (#16670)

    `add_class_tvars` correctly instantiates type variables in the return
    type for class methods but not for static methods. Check if the analyzed
    member is a static method in `analyze_class_attribute_access` and
    substitute the type variable in the return type in `add_class_tvars`
    accordingly.

    Fixes #16668.

Thanks, @kourbou!

Is it worth adding a test for this case?