-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Higher-Kinded TypeVars #548
Comments
I think this came up few times in other discussions, for example one use case is python/mypy#4395. But TBH this is low priority, since such use cases are quite rare. |
damn, I searched very thoroughly but did not find this one! 😄 |
Adding your +1 is all fine, but who's going to do the (probably pretty
complicated) implementation work?
|
are you approving the feature? |
I am neither approving nor disapproving. Just observing that it may be a
lot of work for marginal benefits in most codebases.
|
awfully pragmatic. Where's your sense of adventure? 😄 |
Similar to microsoft/TypeScript#1213 Not sure if the discussion over there provides any useful insights to the effort over here. |
Hi @tek. I'm also very interested in this, so I'd like to ask if you had any progress with this and volunteer to help if you want. |
@rcalsaverini sorry, I've been migrating my legacy code to haskell and am abandoning python altogether. but I wish you great success! |
Oh, sad to hear but I see your point. Thanks. |
Just to add another use case (which I think relates to this issue): Using Literal types along with overloading It is one step closer to being possible due to the most recent mypy release's support for honoring the return type of Note: this is basically a stripped-down version of Django's Field class: # in stub file
from typing import Generic, Optional, TypeVar, Union, overload, Type
from typing_extensions import Literal
_T = TypeVar("_T", bound="Field")
_GT = TypeVar("_GT")
class Field(Generic[_GT]):
# on the line after the overload: error: Type variable "_T" used with arguments
@overload
def __new__(cls: Type[_T], null: Literal[False] = False, *args, **kwargs) -> _T[_GT]: ...
@overload
def __new__(cls: Type[_T], null: Literal[True], *args, **kwargs) -> _T[Optional[_GT]]: ...
def __get__(self, instance, owner) -> _GT: ...
class CharField(Field[str]): ...
class IntegerField(Field[int]): ...
# etc...
# in code
class User:
f1 = CharField(null=False)
f2 = CharField(null=True)
reveal_type(User().f1) # Expected: str
reveal_type(User().f2) # Expected: Union[str, None] |
I wonder if this is what I need or if there's currently a work around for my (slightly simpler) case?: I'm building an async redis client with proper type hints. I have a "Commands" class with methods for all redis commands ( This is easy enough to implement in python, but not so easy to type hint correctly. Basic example: class Redis:
def execute(self, command) -> Coroutine[Any, Any, Union[None, str, int, float]]:
return self.connection.execute(...)
def get(self, *args) -> Coroutine[Any, Any, str]:
...
return self.execute(command)
def set(self, *args) -> Coroutine[Any, Any, None]:
...
return self.execute(command)
def exists(self, *args) -> Coroutine[Any, Any, bool]:
...
return self.execute(command)
# ... and many MANY more ...
class RedisPipeline(Redis):
def execute(self, command) -> None:
self.pipeline.append(command) I tried numerous options to make Is there any way around this with python 3.8 and latest mypy? If not a solution would be wonderful - as far as I can think, my only other route for proper types is a script which copy and pastes the entire class and changes the return types in code. |
@samuelcolvin I don't think this question belongs in this issue. The reason for the failure (knowing nothing about Redis but going purely by the code you posted) is that in order to make this work, the base class needs to switch to an
|
I get that, but I need all the public methods to definitely return a coroutine. Otherwise, if it returned an optional coroutine, it would be extremely annoying to use. What I'm trying to do is modify the return type of many methods on the sub-classes, including "higher kind" types which are parameterised. Hence thinking it related to this issue. |
Honestly I have no idea what higher-kinded type vars are -- my eyes glaze over when I hear that kind of talk. :-) I have one more suggestion, then you're on your own. Use a common base class that has an |
Okay, so the simple answer is that what I'm trying to do isn't possible with python types right now. Thanks for helping - at least I can stop my search. |
I suspect that the reason is that it simply isn't type-safe, and you
couldn't do it (using subclassing) in any other typed language either.
|
humm, but the example above under "Basic example" I would argue IS type-safe. All the methods which end Thus I don't see how this as any more "unsafe" than normal use of generics. |
@gvanrossum, I can relate! I wonder if bidict provides a practical example of how this issue prevents expressing a type that you can actually imagine yourself needing. >>> element_by_atomicnum = bidict({0: "hydrogen", 1: "helium"})
>>> reveal_type(element_by_atomicnum) # bidict[int, str]
# So far so good, but now consider the inverse:
>>> element_by_atomicnum.inverse
bidict({"hydrogen": 0, "helium": 1}) What we want is for mypy to know this: >>> reveal_type(element_by_atomicnum.inverse) # bidict[str, int] merely from a type hint that we could add to a super class. It would parameterize not just the key type and the value type, but also the self type. In other words, something like: KT = TypeVar('KT')
VT = TypeVar('VT')
class BidirectionalMapping(Mapping[KT, VT]):
...
def inverse(self) -> $SELF_TYPE[VT, KT]:
... where |
Okay, I think that example is helpful. I recreated it somewhat simpler (skipping the inheritance from from abc import abstractmethod
from typing import *
T = TypeVar('T')
KT = TypeVar('KT')
VT = TypeVar('VT')
class BidirectionalMapping(Generic[KT, VT]):
@abstractmethod
def inverse(self) -> BidirectionalMapping[VT, KT]:
...
class bidict(BidirectionalMapping[KT, VT]):
def __init__(self, key: KT, val: VT):
self.key = key
self.val = val
def inverse(self) -> bidict[VT, KT]:
return bidict(self.val, self.key)
b = bidict(3, "abc")
reveal_type(b) # bidict[int, str]
reveal_type(b.inverse()) # bidict[str, int] This passes but IIUC you want the ABC to have a more powerful type. I guess here we might want to write it as def inverse(self: T) -> T[VT, KT]: # E: Type variable "T" used with arguments Have I got that? |
Exactly! It should be possible to e.g. subclass bidict (without overriding inverse), and have mypy realize that calling inverse on the subclass gives an instance of the subclass (with the key and value types swapped as well). This isn’t only hypothetically useful, it’d really be useful in practice for the various subclasses in the bidict library where this actually happens (frozenbidict, OrderedBidict, etc.). Glad this example was helpful! Please let me know if there’s anything further I can do to help here, and (can’t help myself) thanks for creating Python, it’s such a joy to use. |
Ah, so the And now I finally get the connection with the comment that started this issue. But I still don't get the connection with @samuelcolvin's RedisPipeline class. :-( |
I would also say that this example is really simple, common, but not supported: def create(klass: Type[T], value: K) -> T[K]:
return klass(value) We use quite a lot of similar constructs in As a workaround I am trying to build a plugin with emulated HKT, just like in some other languages where support of it is limited. Like:
Paper on "Lightweight higher-kinded polymorphism": https://www.cl.cam.ac.uk/~jdy22/papers/lightweight-higher-kinded-polymorphism.pdf TLDR: So, instead of writing |
Hey! Yeah, sorry, I messed up sync with the upstream so had to reclone. I'll push the draft shortly. |
Awesome, thanks! To be completely honest, theoretical typing discussions are... a bit too abstract for me at times. So I'm not 100% sure that my use case above is actually and example of HKT. But if it is, feel free to use it! |
I think libraries like einops would also benefit greatly from this. It would be pretty useful to be able to get shape information both into and out of any sort of array manipulation function. |
I'm not sure if further motivating examples are useful, but here's a simple function, not involved in type checking, that I don't believe can be described today: # sketch -- not carefully checked
T = TypeVar('T')
C = TypeVar('C', bound='Collection[T]')
def filter_and_count_dups(c: C[T]) -> Tuple[C[T], Counter[T]]:
'''copy any 'Collection' w/o duplicates, counting what's omitted
Args:
c: any collection of elements
Returns:
(c_wo_dups, dup_counts), where...
c_wo_dups: a copy of 'c', with the 2nd and subsequent
appearances of any given element omitted
dup_counts: counts removed duplicates
'''
seen: Set[T] = set()
ret_list: List[T] = []
ret_counter = Counter[T]()
for elem in c:
if elem in seen:
ret_counter[elem] += 1
else:
ret_list.append(elem)
seen.add(elem)
ret_wo_dups = type(c)(ret_list)
return (ret_wo_dups, ret_counter)
r1 = filter_and_count_dups((1, 2, 4, 3, 4, 4, 2, 1))
reveal_type(r1) # ≈ Tuple[Tuple[int, ...], Counter[int]]
print(repr(r1)) # ≈ ((1, 2, 4, 3), Counter({4: 2, 2: 1, 1: 1}))
r2 = filter_and_count_dups(list('alphabet'))
reveal_type(r2) # ≈ Tuple[List[str], Counter[str]]
print(repr(r2)) # ≈ (['a', 'l', 'p', 'h', 'b', 'e', 't'], Counter({'a': 1})) |
One more use case, consider the following code: class Handler[I, O](Protocol):
def __call__(self, input: I, /) -> O: ...
class Modify(Protocol):
def modify[O](self, f: Callable[[Self], O], /) -> O: ...
# Mixing styles since idk how to properly express it in 695
M = TypeVar("M", bound="Modify")
class Ext[Inner]:
inner: Inner
def modify[O](self: Ext[M], f: Callable[[M], O], /) -> Ext[O]:
return Ext(f(self.inner)) I can extend class MyExt[Inner](Ext[Inner]):
def another_method(self) -> MyExt[SomeOtherType]:
return self.modify(xxx) However, since class Ext[Inner]:
def modify[O](self: Self[M], f: Callable[[M], O]) -> Self[O]:
return type(self)(self.inner.modify(f)) I have encountered that limitation when writing syntactic sugar for my library for handlers. class Predicate[I](Handler[I, bool], Protocol):
... Here, we specialize class PredicateExt[P](Ext[P]):
def and_[I](self: PredicateExt[Predicate[I]], rhs: Predicate[I]) -> PredicateExt[Predicate[I]]:
return self.modify(lambda lhs: And(lhs, rhs)) ... but there's no way to do that without triggering type-checker, unless we had HKTs |
I'm starting to think that by simply making With this, we could write some HKT function from collections.abc import Sequence
from types import GenericAlias
def f[T: Sequence](x: GenericAlias[T, int], /) -> GenericAlias[T, str]: ... Note that type-checkers might complain about the missing type argument in So to be a little bit more specific, the generic class GenericAlias[T, *Ps]:
@property
def __origin__(self) -> type[T]: ...
@property
def __args__(self) -> tuple[*Ps]: ...
... # etc Note that because a variadic type-parameter like Through the But there's one big problem with this approach: >>> type(list[str])
<class 'types.GenericAlias'>
>>> class Spam[T]: ...
...
>>> type(Spam[str])
<class 'typing._GenericAlias'> There are two generic aliases! |
Hi. Here's the code example I have from my project
This obviously does not work. mypy requires I'd suggest the syntax for the feature to be something as simple as
Cheers. |
Can you be more specific?
Since Python 3.12 the new PEP 695 generic syntax is preferred over |
Someone pointed me to this issue. I just crashed hard onto this one today. I am writing generic tree algorithms. I have: T = TypeVar('T')
# What every node should have in common
class BinaryNode(Protocol[T]): ...
# Represents the actual node that is used by e.g. AVLTree
N = TypeVar('N', bound=BinaryNode)
# Parameterized on the actual node that is used and the value of it
class BinaryTree(Generic[N[T], T]):
def rotate_left(self, node: N[T]) -> None: ...
def add(self, value: T) -> None: ... However, If anyone knows workarounds for this pattern I'd be happy to learn them. |
I've been able to work around this issue using default type parameters in python 3.13. The idea is to have both a node and a T param like so # What every node should have in common
class BinaryNode[T]: ...
# Parameterized on the actual node that is used and the value of it
class BinaryTree[T, N: BinaryNode[Any] = BinaryNode[T]]:
def rotate_left[_T, _N: BinaryNode[Any] = BinaryNode[_T]](self:BinaryTree[_T, _N], node: _N) -> None: ...
def add(self, value: T) -> None: ... Note that this is subtlety broken, as the bound on BinaryNode[T] is For a working example (applied to a similar array type): |
I think I may have a case that might be related to this: class Language(Enum):
...
LanguageT = TypeVar('LanguageT', bound=Language)
class Params(Generic[LanguageT]):
target_language: LanguageT
ParamsT = TypeVar('ParamsT', bound=Params)
class Tasks(Generic[LanguageT, ParamsT]): # I'd like it to be Generic[LanguageT, ParamsT[LanguageT]]
source_language: LanguageT
params: list[ParamsT] # I'd like it to be list[ParamsT[LanguageT]] I then have several providers which all have their own class ProviderALanguage(Language):
...
class ProviderAParams(Params[ProviderALanguage]):
...
class ProviderBLanguage(Language):
...
class ProviderBParams(Params[ProviderBLanguage]):
... Ideally I'd need the type-checker to catch bad cases of inheritance as well as wrongful usage: class ProviderATasks(Tasks[ProviderALanguage, ProviderBParams]):
...
tasks = Tasks[ProviderALanguage, ProviderBParams](...) Currently I don't see a way to do this, but I think this proposed feature would help. It's also possible I can't see the obvious, as I've been looking at this so long... |
The best you could currently do is (using Python 3.12 syntax, but this is also possible with class Language(Enum): ...
class Params[LanguageT: Language]:
target_language: LanguageT
class Tasks[LanguageT: Language]:
source_language: LanguageT
params: list[Params[LanguageT]] edit: alternatively, you could also parametrize the entire params list like class Language(Enum): ...
class Params[LanguageT: Language]:
target_language: LanguageT
class Tasks2[LanguageT: Language, ParamsListT: list[Params[Language]]]:
source_language: LanguageT
params: ParamsListT this type Tasks[LanguageT: Language] = Tasks[LanguageT, list[Params[LanguageT]]] |
I thought of that, but it doesn't work for me as I really need
This feels like a more convoluted way of writing the first example (maybe I'm wrong, though). The type-checker (and the code completion tools) still assume t = Tasks[LangAImpl]()
t.params.append(ParamsAImpl())
t.params[0]. # any ParamsAImpl-specific attributes are not shown by code-completion tools Thanks for trying to help. I'm trying to have it all, I know. I should have been more clear in that I can't get rid of the |
Exactly; I'm not trying to say that it's a solution, but that without HKT, it's the best we can currently do 🤷🏻 |
Ok then, here's some extra syntax. def my_fn[T_elm: (int, float, str), T_seq: list[T_elm] | tuple[T_elm]](
seq: T_seq[T_elm],
) -> T_seq[T_elm]:
return seq which nowadays fails
But this syntax is actually covered in PEP 695 # The following generates no compiler error, but a type checker
# should generate an error because an upper bound type must be concrete,
# and ``Sequence[S]`` is generic. Future extensions to the type system may
# eliminate this limitation.
class ClassA[S, T: Sequence[S]]: ...
# The following generates no compiler error, because the bound for ``S``
# is lazily evaluated. However, type checkers should generate an error.
class ClassB[S: Sequence[T], T]: ... So it's syntax that has been suggested but not yet supported or precised. Or we could allow the syntax I showed above, where TypeVar can be subscripted like a generic. T_elm = TypeVar("T_elm", float, int, str)
T_seq = TypeVar[T_elm]("T_seq", list[T_elm], tuple[T_elm]) |
I like this 👍🏻
This is confusing when doing "meta-typing" of |
aka type constructors, generic TypeVars
Has there already been discussion about those?
I do a lot of FP that results in impossible situations because of this. Consider an example:
I haven't found a way to make this work, does anyone know a trick or is it impossible?
If not, consider the syntax as a proposal.
Reference implementations would be Haskell, Scala.
optimally, the HK's type param would be indexable as well, allowing for
F[X[X, X], X[X]]
Summary of current status (by @smheidrich, 2024-02-08):
peps
repo. The stub PEP draft so far contains a few examples of the proposed syntax.The text was updated successfully, but these errors were encountered: