Add support for Self type (#14041)

Ref #12840 
Fixes #11871
Fixes #14089

This is an alternative implementation to two existing PRs:
#11666,
#13133. This PR treats `typing.Self`
as pure syntactic sugar, and transforms it into a type variable early
during semantic analyzis.

This way we can re-use all the existing machinery and handled edge cases
for self-types. The only new thing is self-type for _attributes_ (as
proposed in the PEP). This required handling in several places, since
attribute access is duplicated in several places (see #7724), plus
special forms (like NamedTuples and TypedDicts) and dataclasses plugin
require additional care, since they use attribute annotations in special
ways.

I don't copy all the existing tests for "old style" self-types, but only
some common use cases, possible error conditions, and relevant new edge
cases, such as e.g. special forms mentioned above, and implicit type
variable binding for callable types.

docs/source/error_code_list2.rst

@@ -82,6 +82,25 @@ Example:
         # Error: Redundant cast to "int"  [redundant-cast]
         return cast(int, x)
 
+Check that methods do not have redundant Self annotations [redundant-self]
+--------------------------------------------------------------------------
+
+Such annotations are allowed by :pep:`673` but are redundant, so if you want
+warnings about them, enable this error code.
+
+Example:
+
+.. code-block:: python
+
+   # mypy: enable-error-code="redundant-self"
+
+   from typing import Self
+
+   class C:
+       # Error: Redundant Self annotation on method first argument
+       def copy(self: Self) -> Self:
+           return type(self)()
+
 Check that comparisons are overlapping [comparison-overlap]
 -----------------------------------------------------------
 

docs/source/generics.rst

@@ -264,15 +264,8 @@ Generic methods and generic self
 You can also define generic methods — just use a type variable in the
 method signature that is different from class type variables. In particular,
 ``self`` may also be generic, allowing a method to return the most precise
-type known at the point of access.
-
-.. note::
-
-   This feature is experimental. Checking code with type annotations for self
-   arguments is still not fully implemented. Mypy may disallow valid code or
-   allow unsafe code.
-
-In this way, for example, you can typecheck chaining of setter methods:
+type known at the point of access. In this way, for example, you can typecheck
+chaining of setter methods:
 
 .. code-block:: python
 
@@ -333,8 +326,69 @@ or a deserialization method returns the actual type of self. Therefore
 you may need to silence mypy inside these methods (but not at the call site),
 possibly by making use of the ``Any`` type.
 
+Note that this feature may accept some unsafe code for the purpose of
+*practicality*. For example:
+
+.. code-block:: python
+
+   from typing import TypeVar
+
+   T = TypeVar("T")
+   class Base:
+       def compare(self: T, other: T) -> bool:
+           return False
+
+   class Sub(Base):
+       def __init__(self, x: int) -> None:
+           self.x = x
+
+       # This is unsafe (see below), but allowed because it is
+       # a common pattern, and rarely causes issues in practice.
+       def compare(self, other: Sub) -> bool:
+           return self.x > other.x
+
+   b: Base = Sub(42)
+   b.compare(Base())  # Runtime error here: 'Base' object has no attribute 'x'
+
 For some advanced uses of self-types see :ref:`additional examples <advanced_self>`.
 
+Automatic self types using typing.Self
+**************************************
+
+The patterns described above are quite common, so there is a syntactic sugar
+for them introduced in :pep:`673`. Instead of defining a type variable and
+using an explicit ``self`` annotation, you can import a magic type ``typing.Self``
+that is automatically transformed into a type variable with an upper bound of
+current class, and you don't need an annotation for ``self`` (or ``cls`` for
+class methods). The above example can thus be rewritten as:
+
+.. code-block:: python
+
+   from typing import Self
+
+   class Friend:
+       other: Self | None = None
+
+       @classmethod
+       def make_pair(cls) -> tuple[Self, Self]:
+           a, b = cls(), cls()
+           a.other = b
+           b.other = a
+           return a, b
+
+   class SuperFriend(Friend):
+       pass
+
+   a, b = SuperFriend.make_pair()
+
+This is more compact than using explicit type variables, plus additionally
+you can use ``Self`` in attribute annotations, not just in methods.
+
+.. note::
+
+   To use this feature on versions of Python before 3.11, you will need to
+   import ``Self`` from ``typing_extensions`` version 4.0 or newer.
+
 .. _variance-of-generics:
 
 Variance of generic types
@@ -548,7 +602,7 @@ Note that class decorators are handled differently than function decorators in
 mypy: decorating a class does not erase its type, even if the decorator has
 incomplete type annotations.
 
-Suppose we have the following decorator, not type annotated yet, 
+Suppose we have the following decorator, not type annotated yet,
 that preserves the original function's signature and merely prints the decorated function's name:
 
 .. code-block:: python

docs/source/more_types.rst

@@ -804,9 +804,10 @@ classes are generic, self-type allows giving them precise signatures:
 .. code-block:: python
 
    T = TypeVar('T')
-   Q = TypeVar('Q', bound='Base[Any]')
 
    class Base(Generic[T]):
+       Q = TypeVar('Q', bound='Base[T]')
+
        def __init__(self, item: T) -> None:
            self.item = item
 

mypy/checker.py

@@ -39,7 +39,7 @@
 from mypy.erasetype import erase_type, erase_typevars, remove_instance_last_known_values
 from mypy.errorcodes import TYPE_VAR, UNUSED_AWAITABLE, UNUSED_COROUTINE, ErrorCode
 from mypy.errors import Errors, ErrorWatcher, report_internal_error
-from mypy.expandtype import expand_type, expand_type_by_instance
+from mypy.expandtype import expand_self_type, expand_type, expand_type_by_instance
 from mypy.join import join_types
 from mypy.literals import Key, literal, literal_hash
 from mypy.maptype import map_instance_to_supertype
@@ -2488,6 +2488,10 @@ class C(B, A[int]): ...  # this is unsafe because...
                 second_sig = self.bind_and_map_method(second, second_type, ctx, base2)
                 ok = is_subtype(first_sig, second_sig, ignore_pos_arg_names=True)
         elif first_type and second_type:
+            if isinstance(first.node, Var):
+                first_type = expand_self_type(first.node, first_type, fill_typevars(ctx))
+            if isinstance(second.node, Var):
+                second_type = expand_self_type(second.node, second_type, fill_typevars(ctx))
             ok = is_equivalent(first_type, second_type)
             if not ok:
                 second_node = base2[name].node
@@ -3068,6 +3072,8 @@ def lvalue_type_from_base(
         if base_var:
             base_node = base_var.node
             base_type = base_var.type
+            if isinstance(base_node, Var) and base_type is not None:
+                base_type = expand_self_type(base_node, base_type, fill_typevars(expr_node.info))
             if isinstance(base_node, Decorator):
                 base_node = base_node.func
                 base_type = base_node.type

mypy/checkexpr.py

@@ -2667,6 +2667,10 @@ def analyze_ordinary_member_access(self, e: MemberExpr, is_lvalue: bool) -> Type
 
             if isinstance(base, RefExpr) and isinstance(base.node, MypyFile):
                 module_symbol_table = base.node.names
+            if isinstance(base, RefExpr) and isinstance(base.node, Var):
+                is_self = base.node.is_self
+            else:
+                is_self = False
 
             member_type = analyze_member_access(
                 e.name,
@@ -2680,6 +2684,7 @@ def analyze_ordinary_member_access(self, e: MemberExpr, is_lvalue: bool) -> Type
                 chk=self.chk,
                 in_literal_context=self.is_literal_context(),
                 module_symbol_table=module_symbol_table,
+                is_self=is_self,
             )
 
             return member_type

mypy/checkmember.py

@@ -6,7 +6,7 @@
 
 from mypy import meet, message_registry, subtypes
 from mypy.erasetype import erase_typevars
-from mypy.expandtype import expand_type_by_instance, freshen_function_type_vars
+from mypy.expandtype import expand_self_type, expand_type_by_instance, freshen_function_type_vars
 from mypy.maptype import map_instance_to_supertype
 from mypy.messages import MessageBuilder
 from mypy.nodes import (
@@ -37,6 +37,7 @@
     erase_to_bound,
     function_type,
     make_simplified_union,
+    supported_self_type,
     tuple_fallback,
     type_object_type_from_function,
 )
@@ -90,6 +91,7 @@ def __init__(
         self_type: Type | None,
         module_symbol_table: SymbolTable | None = None,
         no_deferral: bool = False,
+        is_self: bool = False,
     ) -> None:
         self.is_lvalue = is_lvalue
         self.is_super = is_super
@@ -101,6 +103,7 @@ def __init__(
         self.chk = chk
         self.module_symbol_table = module_symbol_table
         self.no_deferral = no_deferral
+        self.is_self = is_self
 
     def named_type(self, name: str) -> Instance:
         return self.chk.named_type(name)
@@ -152,6 +155,7 @@ def analyze_member_access(
     self_type: Type | None = None,
     module_symbol_table: SymbolTable | None = None,
     no_deferral: bool = False,
+    is_self: bool = False,
 ) -> Type:
     """Return the type of attribute 'name' of 'typ'.
 
@@ -187,6 +191,7 @@ def analyze_member_access(
         self_type=self_type,
         module_symbol_table=module_symbol_table,
         no_deferral=no_deferral,
+        is_self=is_self,
     )
     result = _analyze_member_access(name, typ, mx, override_info)
     possible_literal = get_proper_type(result)
@@ -682,12 +687,12 @@ def analyze_descriptor_access(descriptor_type: Type, mx: MemberContext) -> Type:
     return inferred_dunder_get_type.ret_type
 
 
-def is_instance_var(var: Var, info: TypeInfo) -> bool:
+def is_instance_var(var: Var) -> bool:
     """Return if var is an instance variable according to PEP 526."""
     return (
         # check the type_info node is the var (not a decorated function, etc.)
-        var.name in info.names
-        and info.names[var.name].node is var
+        var.name in var.info.names
+        and var.info.names[var.name].node is var
         and not var.is_classvar
         # variables without annotations are treated as classvar
         and not var.is_inferred
@@ -722,12 +727,16 @@ def analyze_var(
             mx.msg.read_only_property(name, itype.type, mx.context)
         if mx.is_lvalue and var.is_classvar:
             mx.msg.cant_assign_to_classvar(name, mx.context)
+        if not (mx.is_self or mx.is_super) or supported_self_type(
+            get_proper_type(mx.original_type)
+        ):
+            typ = expand_self_type(var, typ, mx.original_type)
         t = get_proper_type(expand_type_by_instance(typ, itype))
         result: Type = t
         typ = get_proper_type(typ)
         if (
             var.is_initialized_in_class
-            and (not is_instance_var(var, info) or mx.is_operator)
+            and (not is_instance_var(var) or mx.is_operator)
             and isinstance(typ, FunctionLike)
             and not typ.is_type_obj()
         ):
@@ -945,7 +954,12 @@ def analyze_class_attribute_access(
             #         x: T
             #     C.x  # Error, ambiguous access
             #     C[int].x  # Also an error, since C[int] is same as C at runtime
-            if isinstance(t, TypeVarType) or has_type_vars(t):
+            # Exception is Self type wrapped in ClassVar, that is safe.
+            if node.node.info.self_type is not None and node.node.is_classvar:
+                exclude = node.node.info.self_type.id
+            else:
+                exclude = None
+            if isinstance(t, TypeVarType) and t.id != exclude or has_type_vars(t, exclude):
                 # Exception: access on Type[...], including first argument of class methods is OK.
                 if not isinstance(get_proper_type(mx.original_type), TypeType) or node.implicit:
                     if node.node.is_classvar:
@@ -958,6 +972,7 @@ def analyze_class_attribute_access(
             # In the above example this means that we infer following types:
             #     C.x -> Any
             #     C[int].x -> int
+            t = get_proper_type(expand_self_type(node.node, t, itype))
             t = erase_typevars(expand_type_by_instance(t, isuper))
 
         is_classmethod = (is_decorated and cast(Decorator, node.node).func.is_class) or (

mypy/errorcodes.py

@@ -186,6 +186,12 @@ def __str__(self) -> str:
     "General",
     default_enabled=False,
 )
+REDUNDANT_SELF_TYPE = ErrorCode(
+    "redundant-self",
+    "Warn about redundant Self type annotations on method first argument",
+    "General",
+    default_enabled=False,
+)
 
 
 # Syntax errors are often blocking.

mypy/expandtype.py

@@ -2,7 +2,7 @@
 
 from typing import Iterable, Mapping, Sequence, TypeVar, cast, overload
 
-from mypy.nodes import ARG_STAR
+from mypy.nodes import ARG_STAR, Var
 from mypy.types import (
     AnyType,
     CallableType,
@@ -383,3 +383,20 @@ def expand_unpack_with_variables(
             raise NotImplementedError(f"Invalid type replacement to expand: {repl}")
     else:
         raise NotImplementedError(f"Invalid type to expand: {t.type}")
+
+
+@overload
+def expand_self_type(var: Var, typ: ProperType, replacement: ProperType) -> ProperType:
+    ...
+
+
+@overload
+def expand_self_type(var: Var, typ: Type, replacement: Type) -> Type:
+    ...
+
+
+def expand_self_type(var: Var, typ: Type, replacement: Type) -> Type:
+    """Expand appearances of Self type in a variable type."""
+    if var.info.self_type is not None and not var.is_property:
+        return expand_type(typ, {var.info.self_type.id: replacement})
+    return typ

mypy/message_registry.py

@@ -238,6 +238,7 @@ def with_additional_msg(self, info: str) -> ErrorMessage:
     "variable"
 )
 CLASS_VAR_WITH_TYPEVARS: Final = "ClassVar cannot contain type variables"
+CLASS_VAR_WITH_GENERIC_SELF: Final = "ClassVar cannot contain Self type in generic classes"
 CLASS_VAR_OUTSIDE_OF_CLASS: Final = "ClassVar can only be used for assignments in class body"
 
 # Protocol

mypy/nodes.py

@@ -2813,6 +2813,7 @@ class is generic then it will be a type constructor of higher kind.
         "has_type_var_tuple_type",
         "type_var_tuple_prefix",
         "type_var_tuple_suffix",
+        "self_type",
     )
 
     _fullname: Bogus[str]  # Fully qualified name
@@ -2953,6 +2954,9 @@ class is generic then it will be a type constructor of higher kind.
     # in case we are doing multiple semantic analysis passes.
     special_alias: TypeAlias | None
 
+    # Shared type variable for typing.Self in this class (if used, otherwise None).
+    self_type: mypy.types.TypeVarType | None
+
     FLAGS: Final = [
         "is_abstract",
         "is_enum",
@@ -3005,6 +3009,7 @@ def __init__(self, names: SymbolTable, defn: ClassDef, module_name: str) -> None
         self.is_newtype = False
         self.is_intersection = False
         self.metadata = {}
+        self.self_type = None
 
     def add_type_vars(self) -> None:
         self.has_type_var_tuple_type = False
@@ -3222,6 +3227,7 @@ def serialize(self) -> JsonDict:
             "metadata": self.metadata,
             "slots": list(sorted(self.slots)) if self.slots is not None else None,
             "deletable_attributes": self.deletable_attributes,
+            "self_type": self.self_type.serialize() if self.self_type is not None else None,
         }
         return data
 
@@ -3278,6 +3284,8 @@ def deserialize(cls, data: JsonDict) -> TypeInfo:
         ti.slots = set(data["slots"]) if data["slots"] is not None else None
         ti.deletable_attributes = data["deletable_attributes"]
         set_flags(ti, data["flags"])
+        st = data["self_type"]
+        ti.self_type = mypy.types.TypeVarType.deserialize(st) if st is not None else None
         return ti