metaclasses implementing __getitem__ work only in reveal_type(), cannot be used anywhere

[raphCode]

Bug Report

#2827 added support for metaclasses implementing __getitem__(). This can be used for custom generic classes, without inheriting from Generic.
While the functionality is working in reveal_type(), it cannot be used anywhere, since that throws errors again. This renders custom generics unuseable at the moment.

To Reproduce

I took the test case from the PR and added two lines afterwards that try to use the custom generic class.

Playground

from typing import *

class M(type):
    def __getitem__(self, key) -> int: return 1

class A(metaclass=M): pass

reveal_type(A[M])

t: TypeAlias = A[M]
i: A[M] = 2

Expected Behavior

No errors.

Actual Behavior

main.py:8: note: Revealed type is "builtins.int"
main.py:10: error: "A" expects no type arguments, but 1 given  [type-arg]
main.py:11: error: "A" expects no type arguments, but 1 given  [type-arg]
main.py:11: error: Incompatible types in assignment (expression has type "int", variable has type "A")  [assignment]

Your Environment

• Mypy version used: 1.2.0
• Python version used: 3.11

Additional Comments:
This was already reported previously:

• mypy does not handle __class_getitem__ #11501 (comment) (did not get any attention)
• Is it possible to define a subscriptable class which can take type arguments and ellipses? #8414 (the underlying problem was probably not worked out enough)

[cainmagi]

Please take a look at this:

microsoft/pyright#5489 (comment)

I have submitted a similar issue to another type checker, pyright. However, my request was rejected due to the abuse of the static type-hints. It seems that the __getitem__ of a metaclass is deliberately forbidden by the type checker. I guess your request may be disapproved by mypy either, as I had encountered in the pyright issue.

Considering this situation, I have come up with an idea to "fool" the type checker without harming the rules of forbidding __getitem__. The following codes will behave differently in static type-checking and run-time:

from typing import TypeVar, Generic, Sequence, Type, cast
from typing_extensions import reveal_type

T = TypeVar("T")


class Proxy(Generic[T]):
    def __init__(self, objtype: Type[T]) -> None:
        self.objtype = objtype


def proxy(objtype: Type[T]) -> Type[T]:
    return cast(Type[T], Proxy(objtype))


proxy_int = proxy(int)
proxy_str = proxy(str)


# No errors are raised, and I am happy to see that!
def test_func(val1: Sequence[proxy_int], val2: Sequence[proxy_str]) -> Sequence[str]:
    return val2


reveal_type(test_func)  # (val1: Sequence[int], val2: Sequence[str]) -> Sequence[str]

This is still not a perfect solution. The only drawback is that we have to set the fake "type aliases" like proxy_int in the above example, which means that it is required to make every "proxy type" as an alias before using them. If you have a better idea for doing such things without using [...] operator, please let me know. Thank you!

[raphCode]

It seems that the __getitem__ of a metaclass is deliberately forbidden by the type checker.

If this indeed were an illegal or unspecified behavior in the type system, I do not understand this comment of Guido van Rossum himself:

Reopening this, since there's an extra wrinkle specifically for __getitem__ -- because X[Y] is used as generic type parameterization, even with elazarg's fix (#2475) the metaclass's __getitem__ is not honored.

Source: #1771 (comment)

To me it looks like he acknowledges that classes with a metaclass implementing __getitem__ can be used as a custom generic type.

[cainmagi]

It seems that the __getitem__ of a metaclass is deliberately forbidden by the type checker.

If this indeed were an illegal or unspecified behavior in the type system, I do not understand this comment of Guido van Rossum himself:

Reopening this, since there's an extra wrinkle specifically for __getitem__ -- because X[Y] is used as generic type parameterization, even with elazarg's fix (#2475) the metaclass's __getitem__ is not honored.

Source: #1771 (comment)

To me it looks like he acknowledges that classes with a metaclass implementing __getitem__ can be used as a custom generic type.

Haha! Hopefully, your request can be approved by mypy. I am not a mypy user. This feature is also important to me to implement some "little tricks". pyright used to support this feature, but in the newest version, this feature was reported as a bug and got deliberately disapproved.

If your request has been successfully approved, maybe I can use this issue to try to persuade Eric (maintainer of pyright) to follow up. Good luck!

[erictraut]

It's fine to supply a __getitem__ method in your metaclass and expect MyClass[x] to work within a regular (value) expression. A type checker should allow that, and both mypy and pyright do. However, type checkers should not allow MyClass[int] to work within a type expression (an annotation or type alias definition) because the index operation in this case is not supplying type arguments to a generic class. If your intent is to define a generic class, then your class should derive from typing.Generic.

class Meta(type):
    def __getitem__(self, key) -> int:
        return 1

class A(metaclass=Meta):
    pass

# Legal when used in value expressions
v = A[Meta] + 1  # This is legal
reveal_type(A[Meta])  # This is legal

# Illegal when used in type expressions
t: TypeAlias = A[Meta]  # This is illegal
x: A[Meta] = A()  # This is illegal

Mypy and pyright are consistent here. Both are doing the right thing, IMO, and I don't consider this a bug.

[raphCode]

I have an actual usecase here for custom generics: #15096
Note that a solution is not possible using typing.Generic.

That being said, I find it very much not nice that class subscription works for typing.Generic and enum.Enum, but there is no possibility to use the same feature in custom code.

because the index operation in this case is not supplying type arguments to a generic class.

In my understanding, having a metaclass which defines __getitem__ is the very much definition of a generic class?

[erictraut]

I have an actual usecase here for custom generics

If you have a new use case that isn't supported by today's Python static type system, you're welcome to suggest ways to extend the type system support it. The python/typing discussion forum is a good place for this. I don't know how receptive the typing community would be to such a feature, but it may be worth proposing it to get a sense for whether there is a broad need among typed Python users that could justify adding such a feature. The typing community may also be able to provide suggestions for other ways to solve your problem that fall within the bounds of the current type system features.

In my understanding, having a metaclass which defines getitem is the very much definition of a generic class?

No, the definition of a generic class is that it subclasses from typing.Generic. The fact that typing.Generic happens to use __getitem__ in its underlying implementation is incidental. There are other reasons why a metaclass might want to implement a __getitem__ method, so a static type checker cannot assume that the presence of a __getitem__ method on a metaclass implies that the metaclass is implementing generic semantics. In the code sample at the top of this issue, for example, the __getitem__ method is annotated to return an int, which makes no sense if this were intended to implement a custom generic.

[raphCode]

Alright. I see that mypy and pyright agree: Using custom generics in type annotations does not work currently.

However, I do not see why this has to be the case. Does it violate a PEP?
Is there a specification what can be used in a type expression and what only in value expressions?

The fact that it works in a value expression makes me feel that this is a half-implemented feature or an artifical restriction rather than desired behavior.

[erictraut]

Does it violate a PEP?

PEP 484 introduced typing.Generic and explained that this was the mechanism by which classes should be designated as generic. Here's the text from PEP 484:

You can include a Generic base class to define a user-defined class as generic. ... Generic[T] as a base class defines that the class LoggedVar takes a single type parameter T.

PEP 484 does not provide any other way to define a generic class other than by deriving from Generic. Nor do any other subsequent PEPs, typing documentation, or agreed-upon typing conventions.

As I mentioned, there could be many reasons why a metaclass might want to implement __getitem__. It would be inappropriate for a static type checker to assume that the intent is to implement semantics that somehow shadow the behavior of typing.Generic. In fact, it would be odd to make that assumption because typing.Generic was designed as the way to express that a class is generic. Supporting alternative mechanisms to do this would be odd unless there was a really good justification to do so.

I suspect there are better and simpler solutions to the problem you're trying to solve, but I can't say for sure because I don't fully understand your constraints.

Incidentally, your "attempt 2" in the linked issue (which makes use of class decorators) looks like a good approach to me. It works fine in pyright but unfortunately doesn't work with mypy because of its lack of support for class decorators. Perhaps the mypy maintainers could be convinced to prioritize a fix for this.

[ikonst]

@erictraut My hunch is that OP assumes more of mypy that it is; that is, OP hoped mypy has a generalized mental model where typing.Generic is just a special case of "a class with a __getitem__", that mypy somehow "understands" or is very close to "understanding" it. In reality, generics are very much special-cased, so whatever OP is hoping to achieve is not a low-hanging fruit.

[raphCode]

I think my line of thought originally was like this:
There are some ways of "performing calculations" in the type system:

• class and function decorators can take an existing type / signature and change it into a different one
• TypeVar and TypeVarTuple can capture concrete types. E.g. when used in a function signature, it 'specializes' the function at call sites
• typing.Generic can similarly be specialized for concrete types with the square bracket syntax, e.g. list[int]

The fact that "custom generics" via metaclass __getitem__ happen to look like generics with square brackets made me believe that this feature should be supported all the way through, namely using it in type expressions.
After all, they do already work in reveal_type() as expected! The restriction to value expressions therefore feels pretty arbitrary.

---

I now realize that all these above things are special-cased in type checkers and not about to be supported.
Maybe there is room for improvement in the error messages that explain the distinction better for someone who has not arrived at this insight yet.

As mentioned, my use case in #15096 could also be solved using class decorators, which mypy not yet supports.

[cainmagi]

Hello, @raphCode!

After talking with Eric at microsoft/pyright#5526, I think I have found a legal way to implement what you want to do in this issue.

The solution equivalent to your example is as follows:

from typing import Generic, TypeVar, TYPE_CHECKING
from typing_extensions import TypeAliasType, reveal_type


T = TypeVar("T")
M = TypeAliasType("M", int, type_params=(T,))


if TYPE_CHECKING:
    A = M[T]
    # The following line should be a better definition.
    # A = TypeAliasType("A", M[T], type_params=(T,))
else:

    class A(Generic[T]):
        ...


i: A[M] = 2  # no error is raised

reveal_type(A[M])  # type[int]
reveal_type(i)  # Literal[2]

Note that TypeAliasType is an experimental feature introduced in Python3.12. The above codes are written for maximal compatibility. typing_extensions can support this feature since Python3.7. If using Python3.12, the definition of M should be:

type M[T] = int

The using of TYPE_CHECKING can let the codes behave differently during the type checking and the run-time.

The above codes have been tested to work for Python>=3.7 and pyright without any errors.

I think changing the implementation of M can do anything we want to do with overriding metaclass __getitem__.

Run-time behavior of TypeAliasType

By the way, even if you do not provide the run-time version of A or M, an alias like M can be recognized differently from using the original type in the run time:

import inspect
from typing import Sequence, TypeVar
from typing_extensions import TypeAliasType, reveal_type

T = TypeVar("T")

proxy = TypeAliasType("proxy", T, type_params=(T,))


def test_func(val1: Sequence[proxy[int]]):
    ...


annotation = inspect.signature(test_func).parameters["val1"].annotation.__args__[0]
name = annotation.__origin__.__name__
print(annotation.__args__, name, name.__class__)  # (<class 'int'>,) proxy <class 'str'>

reveal_type(test_func)  # (val1: Sequence[int]) -> None

That would allow us to do a lot of tricks in the run time. However, I have to acknowledge that this should have a sense of abuse. Personally, I think providing a run-time implementation with TYPE_CHECKING is a better way.

Back to Python<3.12?

Note that the type checks related to TypeAliasType has been supported by pyright, but not supported in mypy yet. Before Python3.12, the TypeAlias plays a similar role to TypeAliasType. In other words, the codes can be rewritten like this:

from typing import Union, Generic, TypeVar, TYPE_CHECKING
from typing_extensions import TypeAlias, Never, reveal_type


T = TypeVar("T")
M: TypeAlias = Union[T, Never]


if TYPE_CHECKING:
    A: TypeAlias = M[T]
else:

    class A(Generic[T]):
        ...


i: A[M] = 2

reveal_type(A[M])
reveal_type(i)

This version is expected to have the same behavior of the TypeAliasType implementation. However, currently, pyright-1.1.316 cannot properly recognize the TypeAlias-based implementation, because defining an alias like M: TypeAlias = T is simply treated as another name of T, not a generic type alias of M[...]. Eric has told me that this issue will be fixed in the next release of pyright.

[erictraut]

I think mypy is working correctly in this case. If there's nothing actionable remaining in this issue, can we close it?