support pep 673 Self type

[zzzeek]

Feature

Support the pep-673 Self type

pep is at: https://www.python.org/dev/peps/pep-0673/

a rough idea of it is that this code:

from typing import Self

class Widget:
    def add_thing(self, value: str) -> Self:
        self.value = value
        return self

is quasi-equivalent to:

from typing import TypeVar

TWidget = TypeVar("TWidget", bound="Widget")

class Widget:
    def add_thing(self: TWidget, value: str) -> TWidget:
        self.value = value
        return self

Pitch

the appeal of the pep is that for the very common pattern of method-chained object construction, methods can easily indicate they return an object of the same type upon which the method is being invoked. For example if I made class SubWidget(Widget), the SubWidget.add_thing() method reports SubWidget as the return type automatically. The pattern using TypeVar seems to be more or less equivlant, but is more verbose requiring the declaration of TypeVar objects per class hierarchy as well as that it has to be explicitly present on the "self" parameter in methods.

We are looking for this feature to make our job of integrating typing into SQLAlchemy an easier job. it seems to be a fairly straightforward translation between two idioms.

cc @CaselIT

[erictraut]

FWIW, both pyright and pyre have implemented this feature in its currently-documented form if you want to play with it.

[zzzeek]

FWIW, both pyright and pyre have implemented this feature in its currently-documented form if you want to play with it.

I know :) it's great. I hate when people email SQLAlchemy with a feature request and say " django does it! " :) but yeah, just trying to see if people using mypy can take advantage also in the near term or if there's no roadmap for this one here.

[AlexWaygood]

There is already a draft PR working on adding support for PEP 673: #11666

[zzzeek]

@AlexWaygood we are literally trying to choose right now if SQLAlchemy 2.0, for which we hope to have beta releases by the end of Q1 2022, can use pep673 Self or not. I know this is normally not something projects can estimate but is it your experience a PR like the one suggested in #11666 can move towards being merged within a few months or is it likely it could stall for a long time?

[zzzeek]

at the same time, if we do release 2.0 using pep 673, mypy users would be more interested in seeing this PR completed so for that reason alone I might decide to use it now.

[AlexWaygood]

Hey @zzzeek -- unfortunately, I'm not really the right person to ask about this :// I'm not actually on the mypy core team, and have only contributed a few documentation fixes to mypy -- I tend to contribute more towards the CPython & typeshed repos, I just stalk the mypy issue tracker from time to time 😄 Perhaps @hauntsaninja might be able to shed some more light?

Having said that, I think the approach typeshed has taken would probably serve you well! You could define a single TypeVar in a standalone module like this (let's call it typing_utils):

# typing_utils.py
from typing import TypeVar

Self = TypeVar('Self')

Then, in the rest of your code base, you could use that TypeVar like this:

# foo.py
from typing_utils import Self

class Foo:
    def returns_self(self: Self) -> Self:
        return self
    @classmethod
    def returns_cls(cls: type[Self]) -> type[Self]:
        return cls

In that way, you'd get type annotations that work perfectly for now; and if PEP 673 is accepted and implemented, you could quite easily run a single script using AST to clean up your code base, converting Foo into the following:

# foo.py
from typing_extensions import Self

class Foo:
    def returns_self(self) -> Self:
        return self
    @classmethod
    def returns_cls(cls) -> type[Self]:
        return cls

The old version of Foo (using pre-PEP 673 syntax) and the new version (using PEP 673 syntax) would have semantically identical type annotations from mypy's perspective.

[hauntsaninja]

I also just stalk the mypy issue tracker from time to time :-) I don't think there is anyone who gets paid time to work on mypy, so there isn't a concrete roadmap. In my experience, you can expect a lag time of a couple months from contributor-completes-PR to mypy-release-that-contains-said-PR.

Re PEP 673 specifically:
I haven't looked at the linked PR, but in the abstract PEP 673 doesn't seem like a hard thing to implement.
I'll also note that PEP 673 has not yet been officially accepted, so I'd keep that in mind when thinking about what promises you're comfortable making to SQLAlchemy users. AlexWaygood's suggestion of just using a normal type variable (and ignoring the bound) + codemodding at a later time is a good option to consider.

[zzzeek]

Having said that, I think the approach typeshed has taken would probably serve you well! You could define a single TypeVar in a standalone module like this (let's call it typing_utils):

# typing_utils.py
from typing import TypeVar

Self = TypeVar('Self')

OK, we can certainly go with that, the approach we have at the moment is copied from how pep-673 illustrates it, with individual TypeVars that are each "bound" to the target type. There should be no reason that's needed though, right? that is, the "Self" that's not bound will work just as well?

[AlexWaygood]

For situations like this, the bound argument is, unfortunately, still necessary, yes — in an ideal world, mypy should be able to understand that TypeVars in that situation are implicitly bound, but that hasn't been implemented yet.

It depends whether you're using mypy to check function implementations, or just writing stubs, though. If you're just writing stubs, you can get away with a lot more, and the bound argument is usually not required 🙂

[zzzeek]

this is inline typing in SQLAlchemy for 2.0 where we will no longer use stubs.

[AlexWaygood]

this is inline typing in SQLAlchemy for 2.0 where we will no longer use stubs.

In which case, yeah, you'll often need the bound argument, unfortunately, or you'll find mypy will erroneously complain about things you're doing inside your functions 😕 but, it still may be worth adding some kind of 'tag' to these TypeVar names (such as having the "Self" suffix with all of them) so that you can easily identify them in the future and change them quickly if/when PEP 673 is accepted.

from typing import TypeVar

FooSelf = TypeVar('F', bound='Foo')

class Foo:
    bar: int
    
    def change_bar(self: FooSelf, x: int) -> FooSelf:
        self.bar = x  # mypy erroneously complains about this line if the TypeVar is not bound 
        return self

[zzzeek]

yes that's what we've been doing, thanks for the tips!

[AlexWaygood]

yes that's what we've been doing, thanks for the tips!

No problem!

[akrymski]

This is probably a silly idea, but it would be rather neat if class methods returned Self by default: ie if there's a return statement you have to specify return type or Any is assumed. If there's no return statement, self is returned by default, and type Self is assumed.

class Widget:
    def add_thing(self, value: str):
        self.value = value
        # return self is assumed if no return value specified

__init___ constructors already do this in fact. Off the top of my head, I can't really think of cases where it would be a bad idea to return self by default, and it makes chaining calls kinda fun ;-) Could even apply it to globals:

print("Hello").print(" World!")

[TeamSpen210]

That's really a Python language change, not a Mypy change - type checkers can't affect what occurs at runtime. __init__() doesn't actually return self, it always returns None - and Python will raise an exception if it doesn't. The object being returned is done bytype.__call__() in that case.

[jpgoldberg]

Just pitching what I assume you already know.

All (well, nearly all) of my need for from __future__ import annotations is solved by typing.Self. I wanted 'Self' from to moment I started learning Python some months back. Indeed, I had assumed that it already existed, but I learned that I need to way for 3.11.

I also wanted good type checking. I came across mypy, and have crafted mypi.ini files as a matter of routine. So thank you all for creating and maintaining mypy (including the outstanding documentation).

Anyway, I am sure you see where this is going. I really want mypy to support PEP 673 sooner rather than later. If I actually knew how to work with ASTs in practice, I would offer to help. Again, I suspect that you don't need me nagging to know that there are people who are very much hoping for Self support in mypy. But at least let me say that your work on this is very much appreciated.