Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SelfType or another way to spell "type of self" (or, How to define a copy() function) #1212

Closed
gvanrossum opened this issue Feb 10, 2016 · 25 comments

Comments

@gvanrossum
Copy link
Member

A colleague and I were wondering how to define a copy() method in a base class so that when called on an instance of a subclass it is known that it returns an instance of that subclass. We found the following solution:

T = TypeVar('T')
class Copyable:
    def copy(self: T) -> T:
        return self.__class__()  # type: ignore
class X(Copyable):
    def foo(self):
        pass
x = X()
x.foo()
xx = x.copy()
xx.foo()  # OK

Note that the # type: ignore is required for two reasons: you can't call self.__class__(), and it doesn't like the return type. Try e.g. return self and you get

x.py:5: error: Incompatible return value type: expected T`-1, got x.Copyable

It works in --py2 mode too:

    def copy(self):
        # type: (T) -> T
        return self.__class__()  # type: ignore

UPDATE: This doesn't actually work. :-( See next comment.

@JukkaL
Copy link
Collaborator

JukkaL commented Feb 10, 2016

Hmm if this works it's not by design. I actually get errors when type checking the example:

t.py:11: error: Need type annotation for variable   (on line xx = x.copy())
t.py:12: error: None has no attribute "foo"

To make this work properly we could provide a new kind of type: a "self" type, which is basically the concrete type of self (in a derived class it would refer to the derived class, even if the definition is inherited). It's been used in a bunch of languages, though I'm not sure if any mainstream language has it. To make it work well we'd also need to support calling self.__class__.

We could have something like this:

from typing import SelfType

class Copyable:
    def copy(self) -> SelfType:
        return self.__class__()  # TODO still need to be able to call self.__class__
class X(Copyable):
    def foo(self):
        pass
x = X()
x.foo()
xx = x.copy()
xx.foo()  # OK

SelfType would probably only be usable in a covariant position such as in a return type. self and self.__class__(...) would have the type SelfType.

@gvanrossum
Copy link
Member Author

Oh, I could have sworn that it worked but it was late. :-( So yes, I guess this is two feature requests: SelfType and calling self.__class__.

@gvanrossum gvanrossum changed the title How to define a copy() function -- clever solution! How to define a copy() function Feb 10, 2016
@gvanrossum
Copy link
Member Author

(It seems to work when the class is generic, but who knows what other effects that has.)

@JukkaL
Copy link
Collaborator

JukkaL commented Apr 15, 2016

This was closed accidentally?

@gvanrossum
Copy link
Member Author

Could we make this work without new syntax? The following might have potential:

T = TypeVar('T', bound='Copyable')  # 'Copyable' is a forward ref
class Copyable:
    def copy(self: T) -> T: ...

Mypy would have to special-case the snot out of this, and I don't know if bound='ForwardRef' currently works, but it would avoid a new magic variable.

We could use something similar based on Type[T] (see python/typing#107) for a factory class method:

# Same as above, then
    @classmethod
    def factory(cls: Type[T]) -> T: ...

@refi64
Copy link
Contributor

refi64 commented May 14, 2016

Mypy would have to special-case the snot out of this

Eeeewww...

But, more seriously, why exactly does this need to be special-cased? Other than the forward reference, this seems to generally be what I would think would be normal. Is it because the type passed to bound is the type using the type variable?

@gvanrossum
Copy link
Member Author

gvanrossum commented May 14, 2016 via email

@refi64
Copy link
Contributor

refi64 commented May 14, 2016

Ah, ok. So the "special-casing" is allowing self to be a type variable with bound set to the current type?

@gvanrossum
Copy link
Member Author

The bound may not even be necessary. The key behavior we'd want that's not implemented right now is that if you have a subclass:

class C(Copyable):
    pass
x = C().copy()
y = C.factory()

we want the type of x and y to be C.

@JukkaL
Copy link
Collaborator

JukkaL commented May 16, 2016

This seems to me like it could work. This would save us from having to define a special "self type", and we wouldn't need to update the implementation of typing. A pretty neat idea.

A remaining open issue is enforcing a compatible __init__ signature in subclasses.

Here is a more detailed example of how this could work internally:

T = TypeVar('T', bound=Copyable)
class Copyable:
    def copy(self: T) -> T:
        t = self.__class__   # Type[T]
        return t()   # T -> good

We'd probably have to special case at least these things:

  • Implement Type[T] (which is a prerequisite for this).
  • Allow an annotation like this on self and enforce that the bound is compatible. Propagate the annotated type of self through the body of a method.
  • Substitute a value for the type variable when looking up a method of an instance or class that uses the trick. C().copy should have type like () -> C instead of () -> T.
  • Make self.__class__ produce the right type (probably don't need to special case self here). Maybe also support type(self)?
  • Update overriding compatibility checks to know about this. Any override of copy() would also have to use a type variable for self.
  • Disallow using the type variable in an argument type as they are contravariant, or at least disallow most uses (maybe some could actually be supported).
  • It seems that using this T in a non-covariant position, such as in List[T], of the return type should be rejected.

In phase 2, figure out __init__ signature enforcement. We can leave this out in v1, but there is a risk that this will be tricky to retrofit. It would be nice to have a plausible design for phase 2 when we start working on the first phase.

@rwbarton
Copy link
Contributor

rwbarton commented May 22, 2016

As discussed in python/typing#107, with Type[C] this could also make sense for class methods

class C:
    @classmethod
    def make_some(cls: Type[T], n: int) -> List[T]: ...

If d has type Type[D], and D is a subtype of C, then d.make_some(n) would have type List[D].

@gvanrossum
Copy link
Member Author

Yes, but it would need a type variable.

@rwbarton
Copy link
Contributor

Oops, yes! Corrected.

@drakedevel
Copy link

I recently ran into the factory use-case @gvanrossum suggested in a real application, trying to write something like this:

T = TypeVar('T', bound='Msg')

class Msg:
    ...
    @classmethod
    def parse(cls: Type[T], in_f: Any) -> T:
        params = {}
        for name, type_ in cls.fields:
            params[name] = in_f.read(type_)
        return cls(params)

class MsgA(Msg): ...

msg = MsgA.parse(...)

As expected, this doesn't work. Possible workarounds are to move parse to the top level, and call it with parse(MsgA, ...), or even just to replace @classmethod with @staticmethod and call it with Msg.parse(MsgA, ...). No changes to the type signatures or function bodies are needed. In my opinion that's a good argument for this syntax to work in the @classmethod context!

@gvanrossum
Copy link
Member Author

@elazarg Do you want to give this a try? We'll also need some new words for PEP 484 on the issue (but no new syntax, technically). The idea is as follows:

T = TypeVar('T')
class C:
    @classmethod
    def new(cls: Type[T]) -> T: <make one>
    def copy(self: T) -> T: <make a copy>
class D(C): pass
d = D.new().copy()  # has type D

There are probably some additional implied constraints on the type of cls and self here, they are still consrained by Type[C] and C even if T does not spell out that constraint.

@gvanrossum
Copy link
Member Author

Oh, forgot that the PEP text is ready to go: python/peps#89

Having a working implementation in mypy would help here.

@elazarg
Copy link
Contributor

elazarg commented Sep 28, 2016

I will try.

@gvanrossum
Copy link
Member Author

gvanrossum commented Sep 28, 2016 via email

@elazarg
Copy link
Contributor

elazarg commented Sep 28, 2016

It also requires instantiating the typevars in the signature at member access, and checking for strictly-covariant overriding. We'll see what's more.

@elazarg elazarg mentioned this issue Sep 29, 2016
@elazarg
Copy link
Contributor

elazarg commented Sep 29, 2016

It will be nice to have this declaration in the typing module:

Self = TypeVar('Self', covariant=True) 

Where the bound is documented to be that of the current class (lexically).


I have a feeling that the suggested syntax, although nice and seemingly intuitive, is somewhat misleading. It looks like it is standard type variable, but it must be exactly the type of self, so it must be strictly covariant and cannot be used (bare) as the type of other parameters.

This leaves place for mistakes that can be avoided or have better diagnostics by avoiding the declaration of the type of self. It can also be parametrized:

from typing import Self, TypeVar

T = TypeVar('T')

class MyList(Generic(T)):
    @classmethod
    def new(cls) -> Self: <make one>
    def copy(self) -> Self: <make a copy>
    def twice(self) -> Tuple[Self, Self]: <make two copies>

    def copy_as_mapped(self, mapper: Callable[[T], Bla]) -> Self[Bla]: <make a mapped copy>
    def copy2(self, another: Self) -> Self: ... # Error: using bare Self in parameters is unsafe

It is also somewhat more greppable, although self: \w is OK.

@elazarg
Copy link
Contributor

elazarg commented Sep 29, 2016

(It may also ease the implementation, but that's not really important)

@gvanrossum
Copy link
Member Author

gvanrossum commented Sep 29, 2016 via email

@gvanrossum
Copy link
Member Author

We have this now.

PEP change: python/peps@ada7d35

mypy changes: ba85545 and several followups (e.g. 8a5f463, a90841f)

@kyprifog
Copy link

For anyone else that stumbles on this and is looking for the SelfType application, this helped me:

#3661

JelleZijlstra pushed a commit to python/typeshed that referenced this issue Jun 30, 2020
…4298)

The return type of the BaseException.with_traceback() method [1] is not
specific enough. The return type is guaranteed to be of the same type as
‘self’, which is usually a subclass of BaseException.

In fact, .with_traceback() returns ‘self’:

    try:
        raise ValueError
    except Exception as exc:
        assert exc.with_traceback(None) is exc

Fix the annotation to reflect this using the self-type annotation
technique described in PEP484 [2], which is supported by (at least)
mypy [3].

[1] https://docs.python.org/3/library/exceptions.html#BaseException.with_traceback
[2] https://www.python.org/dev/peps/pep-0484/#annotating-instance-and-class-methods
[3] python/mypy#1212
tomchristie pushed a commit to encode/httpx that referenced this issue Oct 2, 2020
* Allow covariants of __enter__, __aenter__, and @classmethod

The problem we currently have is the return type of classes such as
Client does not allow covariants when subclassing or context manager.
In other words:

```python
class Base:
    def __enter__(self) -> Base:  # XXX
        return self

class Derived(Base):
    ...

with Derived() as derived:
   # The type of derived is Base but not Derived. It is WRONG
    ...
```

There are three approaches to improve type annotations.
1. Just do not type-annotate and let the type checker infer
   `return self`.
2. Use a generic type with a covariant bound
   `_AsyncClient = TypeVar('_AsyncClient', bound=AsyncClient)`
3. Use a generic type `T = TypeVar('T')` or `Self = TypeVar('Self')`

They have pros and cons.
1. It just works and is not friendly to developers as there is no type
   annotation at the first sight. A developer has to reveal its type via
   a type checker. Aslo, documentation tools that rely on type
   annotations lack the type. I haven't found any python docuementation
   tools that rely on type inference to infer `return self`. There are
   some tools simply check annotations.

2. This approach is correct and has a nice covariant bound that adds
   type safety. It is also nice to documentation tools and _somewhat_
   friendly to developers. Type checkers, pyright that I use, always
   shows the the bounded type '_AsyncClient' rather than the subtype.
   Aslo, it requires more key strokes. Not good, not good.

   It is used by `BaseException.with_traceback`
   See https://github.com/python/typeshed/pull/4298/files

3. This approach always type checks, and I believe it _will_ be the
   official solution in the future. Fun fact, Rust has a Self type
   keyword. It is slightly unfriendly to documentation, but is simple to
   implement and easy to understand for developers. Most importantly,
   type checkers love it.

   See python/mypy#1212

But, we can have 2 and 3 combined:

```python
_Base = typing.TypeVar('_Base', bound=Base)

class Base:
   def __enter__(self: _Base) -> _Base:
      return self

class Derive(Base): ...

with Derived() as derived:
   ...  # type of derived is Derived and it's a subtype of Base
```

* revert back type of of SteamContextManager to Response

* Remove unused type definitions

* Add comment and link to PEP484 for clarification

* Switch to `T = TypeVar("T", covariant=True)`

* fixup! Switch to `T = TypeVar("T", covariant=True)`

* Add back bound=xxx in TypeVar

Co-authored-by: Florimond Manca <[email protected]>
WorldStar0126 pushed a commit to WorldStar0126/httpx that referenced this issue Mar 30, 2023
* Allow covariants of __enter__, __aenter__, and @classmethod

The problem we currently have is the return type of classes such as
Client does not allow covariants when subclassing or context manager.
In other words:

```python
class Base:
    def __enter__(self) -> Base:  # XXX
        return self

class Derived(Base):
    ...

with Derived() as derived:
   # The type of derived is Base but not Derived. It is WRONG
    ...
```

There are three approaches to improve type annotations.
1. Just do not type-annotate and let the type checker infer
   `return self`.
2. Use a generic type with a covariant bound
   `_AsyncClient = TypeVar('_AsyncClient', bound=AsyncClient)`
3. Use a generic type `T = TypeVar('T')` or `Self = TypeVar('Self')`

They have pros and cons.
1. It just works and is not friendly to developers as there is no type
   annotation at the first sight. A developer has to reveal its type via
   a type checker. Aslo, documentation tools that rely on type
   annotations lack the type. I haven't found any python docuementation
   tools that rely on type inference to infer `return self`. There are
   some tools simply check annotations.

2. This approach is correct and has a nice covariant bound that adds
   type safety. It is also nice to documentation tools and _somewhat_
   friendly to developers. Type checkers, pyright that I use, always
   shows the the bounded type '_AsyncClient' rather than the subtype.
   Aslo, it requires more key strokes. Not good, not good.

   It is used by `BaseException.with_traceback`
   See https://github.com/python/typeshed/pull/4298/files

3. This approach always type checks, and I believe it _will_ be the
   official solution in the future. Fun fact, Rust has a Self type
   keyword. It is slightly unfriendly to documentation, but is simple to
   implement and easy to understand for developers. Most importantly,
   type checkers love it.

   See python/mypy#1212

But, we can have 2 and 3 combined:

```python
_Base = typing.TypeVar('_Base', bound=Base)

class Base:
   def __enter__(self: _Base) -> _Base:
      return self

class Derive(Base): ...

with Derived() as derived:
   ...  # type of derived is Derived and it's a subtype of Base
```

* revert back type of of SteamContextManager to Response

* Remove unused type definitions

* Add comment and link to PEP484 for clarification

* Switch to `T = TypeVar("T", covariant=True)`

* fixup! Switch to `T = TypeVar("T", covariant=True)`

* Add back bound=xxx in TypeVar

Co-authored-by: Florimond Manca <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants