mypy does not handle __class_getitem__

[jayanthkoushik]

PEP 560 defines __class_getitem__ as a way to enable indexing a class (to define custom generic types for instance), but mypy does not seem to recognize this.

To Reproduce

$ cat test.py
class A:
    def __class_getitem__(cls, item):
        return cls

x = A[int]

$ mypy test.py

Expected Behavior

Success: no issues found in 1 source file

Actual Behavior

test.py:6: error: "A" expects no type arguments, but 1 given
Found 2 errors in 1 file (checked 1 source file)

Environment

• mypy 0.910
• Python 3.9.7
• Darwin arm64

[sobolevn]

Oh wow! I always thought it was possible!

PR with the fix is on its way. I hope, that I won't break things.

[sobolevn]

Ok, I misunderstood your problem a bit at first. Please, let me explain what is going on.

class A:
    def __class_getitem__(cls, item):
        return cls

x = A[int]

Here A[int] is treated as "type application" by mypy. This is the core of how we work with Generic types. Like List[int] or YourGeneric[T]. Moreover, __class_getitem__ was added especially for this use-case. It is designed to be representing generic type args.

Mypy here complains about missing Generic[T] base class in your definition. Because we associate "type application" not with __class_getitem__, but with Generic (which provides this magic method to an object).

What is your use-case for bare __class_getitem__?

[jayanthkoushik]

I was trying to implement a generic-like interface indexed with strings. Something like:

GenCls[“spec”]

where new types are constructed at run time based on the string. If I’m not wrong, this isn’t possible with Generic as a base class, since the index isn’t a type.

Is __class_getitem__ not meant to be used directly?

[jayanthkoushik]

I should add: type checking also breaks if using a metaclass with __getitem__, instead of __class_getitem__. I think it makes sense to treat obj[key] as just regular indexing if obj is not a Generic subclass.

[sobolevn]

This might be a good idea! I will try to send a prototype PR today. We can start a further discussion from there.

[AlexWaygood]

It might be a good idea for this to be fixed in mypy, as the language doesn't prevent you from using __class_getitem__ for non-type-checking purposes.

It may be worth noting, however, that the documentation for __class_getitem__ (newly rewritten by me 😉) does include the following warning regarding __class_getitem__:

Custom implementations of class_getitem() on classes defined outside of the standard library may not be understood by third-party type-checkers such as mypy. Using class_getitem() on any class for purposes other than type hinting is discouraged.

[sobolevn]

@AlexWaygood see my #11558 PR, it has lots of problems. I am not sure that this is actually worth the effort.

[AlexWaygood]

@jayanthkoushikim in what way does "type checking also break if using a metaclass with __getitem__, instead of __class_getitem__" 🙂? This seems to work fine:

from typing import TypeVar, Any
T = TypeVar('T')

class Meta(type):
    def __getitem__(cls: type[T], arg: Any) -> type[T]:
        return cls
        
class Foo(metaclass=Meta): ...
reveal_type(Foo["Does this work?"]) # Revealed type is "Type[__main__.Foo*]"

[jayanthkoushik]

@AlexWaygood Hm, interesting that reveal type works. But assigning Foo["..."] to anything, or using it as a type annotation does not work.

$ cat test.py
from typing import TypeVar, Any
T = TypeVar('T')

class Meta(type):
    def __getitem__(cls: type[T], arg: Any) -> type[T]:
        return cls
    
class Foo(metaclass=Meta): ...
reveal_type(Foo["Does this work?"])
x = Foo["Does this work?"]
y: Foo["Does this work?"]

$ mypy test.py
test.py:9: note: Revealed type is "Type[test.Foo*]"
test.py:10: error: "Foo" expects no type arguments, but 1 given
test.py:10: error: Invalid type comment or annotation
test.py:11: error: "Foo" expects no type arguments, but 1 given
test.py:11: error: Invalid type comment or annotation
Found 5 errors in 1 file (checked 1 source file)

[AlexWaygood]

@jayanthkoushik fair enough!

[jayanthkoushik]

IMO, at least x = Foo[…] should be supported since it does not use any undocumented or unrecommended behavior.

[AlexWaygood]

IMO, at least x = Foo[…] should be supported since it does not use any undocumented or unrecommended behavior.

Yes, I think I agree that this would be very nice to have in the case of metaclass __getitem__. For __class_getitem__, I'm much more ambivalent, since, at the end of the day, it was only ever really intended to be used for classes inside the stdlib.

[sobolevn]

It looks like master (at least) is already capable of understanding this code:

class Meta(type):
    def __getitem__(cls, arg: str) -> 'Meta':
        return cls

class My(metaclass=Meta):
    ...

reveal_type(My[1])
# out/ex.py:8: note: Revealed type is "ex.Meta"
# out/ex.py:8: error: Invalid index type "int" for "Type[My]"; expected type "str"

[dmadisetti]

Yeah, see #2827 and #1771 for the fix and implementation of __getitem__ support in metaclasses.

[jamesbraza]

To add a case to this, it would be cool if we could leverage a custom __class_getitem__ while also retaining the typing benefits of Generic. Something like this:

from typing import ClassVar, Generic, TypeVar

T = TypeVar("T")

class Foo(Generic[T]):
    cls_attr: ClassVar[int]

    def __class_getitem__(cls, item: tuple[int, T]):
        cls.cls_attr = item[0]
        return super().__class_getitem__(item[1])

    def __init__(self, arg: T):
        self.arg = arg

foo = Foo[1, bool](arg=True)
assert foo.cls_attr == 1

[ddanier]

I have created a library to generate partial models for pydantic (https://github.com/team23/pydantic-partial). The basic idea would be to have something like Partial[Foo] to create a copy of Foo where all fields allow None and have a default value of None. This tries to follow what Typescript does, see https://www.typescriptlang.org/docs/handbook/utility-types.html#partialtype for reference.

Currently I am using a function to generate the partial model instead of Partial.__class_getitem__ as mypy does not understand what is going on here. Thus I wanted to add this use case here, as it even strictly follows the original intent of the __class_getitem__ method to be used for typing. So it would be really cool to support this and allow typing helpers to work with __class_getitem__, too.

As an additional note: I'm totally aware that I am generating dynamic types here which will pose additional headaches. I'm currently in the progress of investigating how to "tell" mypy that my current approach creates a valid type (error like: "Unsupported dynamic base class"). But this is a totally different story and may be fixable with creating a mypy plugin (I'm possibly going for this, soon). I just stumbled over this issue and thought to add this idea here - as I really would like Partial[...] to work like Partial<...> in Typescript. ;-)

[Dvergatal]

I'm sorry that I'm asking here, but the question I want to ask is strictly connected to the subject of this issue.

Correct me if I'm wrong, but from what I understood, __class_getitem__ serves for type hinting right? That is what I understood from description which has been recently updated by @AlexWaygood and also PEP 560.

Now question is does it concern generic core abstract classes as well? Because I have this feature request #14537 in which I have given an example for which mypy is returning errors:

multi.py:26: error: Incompatible types in assignment (expression has type "List[str]", variable has type "MyIterable[str]")  [assignment]
multi.py:27: error: Incompatible types in assignment (expression has type "List[List[str]]", variable has type "MyIterable[MyIterable[str]]")  [assignment]
multi.py:40: error: Argument 3 to "get_enum_str" has incompatible type "List[str]"; expected "MyIterable[str]"  [arg-type]
multi.py:41: error: Argument 2 to "get_enum_str_list" has incompatible type "List[str]"; expected "MyIterable[str]"  [arg-type]
multi.py:41: error: Argument 3 to "get_enum_str_list" has incompatible type "List[List[str]]"; expected "MyIterable[MyIterable[str]]"  [arg-type]
multi.py:42: error: Argument 2 to "get_enum_str_list" has incompatible type "str"; expected "MyIterable[str]"  [arg-type]
multi.py:42: error: Argument 3 to "get_enum_str_list" has incompatible type "List[List[str]]"; expected "MyIterable[MyIterable[str]]"  [arg-type]