Proposal: Generalize Callable to be able to specify argument names and kinds
#264
Comments
|
Another thing to consider is keyword-only arguments in Python 3. Keyword-only arguments can be optional or non-optional. Maybe use |
|
@JukkaL Yep seems good. |
|
@sixolet Thankyou for your proposal! def cb_stub(x: T, *args, option: str='verbose') -> int:
raise NotImplementedError
def func(callback: Callable[cb_stub]):
...and Args = [Arg(T, name='x'), StarArg(Any), OptionalArg(str, name='option')]
def func(callback: Callable[Args, int]):
...I think the first one is more readable and allows to quickly grasp the function signature. Also it is not clear from your proposal how to type annotate a decorator that preserves the types of all arguments in I would like to reiterate, function stubs together with @JukkaL what do you think about the magic (variadic) type variable from typing import OtherArgs, TypeVar, Callable
R = TypeVar('R')
def change_ret_type(f: Callable[[OtherArgs], R]) -> Callable[[OtherArgs], int]: ...
def add_initial_int_arg(f: Callable[[OtherArgs], R]) -> Callable[[int, OtherArgs], R]: ...
def fix_initial_str_arg(f: Callable[[str, OtherArgs], R]) -> Callable[[OtherArgs], int]:
def ret(*args, **kwargs):
f('Hello', *args, **kwargs)
return ret |
|
Regarding |
|
It's true that this proposal is similar to your proposal based on
|
|
@ilevkivskyi I agree with @sixolet that |
|
OK, if you want to proceed with this, I would propose to try to make it more concise. Here are some tweaks, apart from already mentioned
Arg('x', T)
Arg('y') # same as Arg('y', Any)
With all these tweaks it will look like this: def fun1(x: List[T], y: int = 0, *args, **kwargs: str) -> None:
...has type Callable[[Arg('x', List[T]), OptArg('y', int), StarArg(), KwArg(str)], None]Second example: def fun2(__s: str, __z: T, *, tmp: str, **kwargs) -> int:
...has type Callable[[str, T, Arg('tmp', str, kw_only=True), KwArg()], int]It looks more concise, and still readable. What do you think? |
|
@ilevkivskyi I like most of that. The only bit I think I disagree with is
And that's because I think in this particular case consistency is more important than brevity. Huh. Can we omit the parens when they're not being used for anything? Why not Callable[[str, T, Arg('tmp', str, kw_only=True), KwArg], int] |
|
(Answering my own question, the round parens remind us it's an argspec not a type. Right.) |
|
@sixolet OK By the way I was thinking a bit about how this will interoperate with variadic generics, and I think the solution is simple: As = TypeVar('As', variadic=True)
Ks = TypeVar('Ks', variadic=True)
Input = Callable[[int, StarArg(As), KwArg(Ks)], None]
Output = Callable[[StarArg(As), KwArg(Ks)], None]
Deco = Callable[[Input], Output]Moreover Input = Callable[[int, *OtherArgs], None]
Output = Callable[*OtherArgs], None]But now it is probably not needed in the specification. So that now I give your proposal a solid +1. |
|
I don't understand what's wrong with something like It will be much easier for type checkers to learn, and the meaning is obvious. Why invent new ways of saying the same thing? |
|
It's not syntactically valid. Lambdas can't have type annotations. |
|
@Lazarg Also, we should be careful to avoid confusing people regarding "how do you write a function" vs "how do you write the type of a function" |
|
The type (the interface) of the function is the thing that appears right after its name - the signature. |
|
Oh. Consequences of using : instead of => ? That's very unfortunate. |
|
How about |
|
@elazarg I suspect we'll be better off and have a clearer language if we steer clear of situations where people can confuse an object of a type (a function) with the type itself (the callable). Here it's not exactly what you're suggesting, though, I understand. You're providing a function as the type parameter to a Callable type to demonstrate what type the callable will be. It might be even more confusing, especially if the function that you're providing as a type parameter produces a very special kind of dict object, and that has nothing to do with the return type of the Callable. |
…thing you can do (#2607) Implements an experimental feature to allow Callable to have any kind of signature an actual function definition does. This should enable better typing of callbacks &c. Initial discussion: python/typing#239 Proposal, v. similar to this impl: python/typing#264 Relevant typeshed PR: python/typeshed#793
|
I think we should turn the finalized proposal into a new PEP, rather than amending PEP 484. Who wants to draft it? |
|
I'd be happy to with some process guidance.
…On Wed, May 3, 2017 at 2:29 PM, Guido van Rossum ***@***.***> wrote:
I think we should turn the finalized proposal into a new PEP, rather than
amending PEP 484. Who wants to draft it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#264 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABjs4fGIX9s0pRk3SOrz08WoeOWYvkCAks5r2PGpgaJpZM4JoC-Y>
.
|
|
PEP 1 describes the general process: https://www.python.org/dev/peps/pep-0001/ I'm not sure if these are 100% up to date, but basically you should create a PR for https://github.com/python/peps and assign it PEP number 9999; a reviewer will assign a number before merging the initial draft. Note that the initial draft usually gets merged as soon as it conforms to the form guidelines -- after that the discussion starts, typically with a post to the python-dev mailing list. |
|
@sixolet
|
|
Are there any plans to make it possible to use a placeholder (like a typevar) for the whole argument list? That's helpful for decorators where the wrapped function signature is preserved; contrived e.g.: A = TypeVar('A') # arguments generic
RT = TypeVar('RT') # Return type
def singleton_list(func: Callable[A, RT]) -> Callable[A, List[RT]):
def wrapper(*args, **kwargs):
return [func(*args, **kwargs)]
return wrapperI used |
|
I proposed something quite similar in python/mypy#3028. I think we want to move forward with something like this, but it needs somebody to actually do the work. |
|
Yes, an overload can be given where a union is expected. But note that if someone will give a union there, it will crash at runtime (while being uncaught statically). The problem is that union is much wider type than an overload, each element of the union will be actually better. |
|
Another (simpler) interesting example is the ambiguity about what should this mean: def fun() -> Callable[[T], T]:
...Currently this creates a non-generic function that returns a generic function. This is probably the right thing to do, but this is too implicit, for each given signature mypy tries to guess what the author of the function means (and I am not sure there are any strict rules). It would be more unambiguous if one could write: def same(x: T) -> T:
...
def fun() -> Callable[same]:
...It is clear now that |
|
What's the status of this? Has anyone taken the initiative to draft a PEP? If none of the core devs have the time, I can distill the discussion here down into a PEP. |
|
What's the status of this? Has anyone taken the initiative to draft a
PEP? If not, I can distill the discussion here down into a PEP.
Sadly we've not had time to work on this. If you're interested in drafting
a PEP that would be great! Hopefully you're bringing some opinions of your
own (just transcribing the conversation here isn't sufficient for a PEP. :-)
|
|
Wait you're saying I have to have original thought? :P But seriously, I've dealt with this lack of expressiveness across a couple different projects now, so I do have some thoughts about how this should work. I'll use your "PEP how-to" links above. Thanks! |
|
Can we please have something like this? Callable[[dict, ...], dict]Which means the first argument to the callable must be a This is quite useful for type annotating wrapper functions. def print_apples(fn: Callable[[dict, ...], dict]) -> Callable[[dict, ...], dict]:
def wrapper(fruits, *args, **kwargs):
print('Apples:', fruits['apples'])
return fn(fruits, *args, **kwargs)
return wrapper
@print_apples
def get(fruits, key):
return fruits[key] |
|
@devxpy Note that you can already express lots of things using callback protocols |
|
@howinator did you end up making this PEP? |
|
@ilevkivskyi This does not seem to work with Am I doing something wrong? from typing import Callable, Any
from typing_extensions import Protocol
class ApplesWrapper(Protocol):
def __call__(self, fruits: dict, *args, **kwargs) -> Any: ...
def print_apples(fn: ApplesWrapper) -> ApplesWrapper:
def wrapper(fruits: dict, *args, **kwargs):
print('Apples:', fruits['apples'])
return fn(fruits, *args, **kwargs)
return wrapper
@print_apples
def get(fruits: dict, key):
return fruits[key] |
|
@Seanny123 Now that we have callback protocols, the PEP will be less valuable (if needed at all). The only missing thing is variadic generics, but this is a separate question, and the problem with them is not who will write the PEP, but who will implement it. @devxpy Well, mypy correctly says that type of I think what you want is |
…thing you can do (#2607) Implements an experimental feature to allow Callable to have any kind of signature an actual function definition does. This should enable better typing of callbacks &c. Initial discussion: python/typing#239 Proposal, v. similar to this impl: python/typing#264 Relevant typeshed PR: python/typeshed#793
|
Should we close this issue now that callback protocols are described in PEP 544 (which was accepted)? |
|
I seem to be in the minority, but I find callback protocols too verbose, and not very intuitive. I'd rather have a decorator you can put on a dummy class ApplesWrapper(Protocol):
def __call__(self, fruits: dict, *args, **kwargs) -> Any: ...I'd prefer @some_decorator
def ApplesWrapper(fruits: dict, *args, **kwargs) -> Any: ...My reasons are hard to explain -- I find the need to use |
|
I agree that the current syntax is overly verbose. On the other hand complex callables don't seem to be required more than occasionally by most programmers. Maybe somebody will come up with a nicer and more concise syntax, though. |
|
To find similarity to another thing within the language, argspec def ApplesWrapper(fruits: dict, *args, **kwargs) -> AnyOr, if we'd like to be even less verbose with an even shorter syntax, making this it's entirely "own" thing. |
|
Maybe we could add in some decorator that when present will promote some function into a callable protocol? Basically, make it so that type checkers treat Guido's second example as a shorthand for the first. The decorator could just be the identity function at runtime. We could also make it literally construct the protocol class if the distinction between a function vs a callable object is important to preserve. At least for mypy, I feel we could pretty easily implement this via the plugin system. |
|
@antonagestam typed def callback(a: str) -> bool:
...
# or
typespec callback(...) -> bool:
...
# or
typedef callback(...) -> bool:
...instead of And then only declare the implementation's intention of fulfilling the contract using a decorator: @typeimpl(callback)
def mycallback(...) -> bool:
...As the language already introduced class variable annotations and return type annotations I'd find a syntax extension more natural for "first class support" of function types than a decorator syntax that leads to typing being implemented counter-intuitively through a mixture of new language keywords and special code. |
|
@jdelic As for naming of this hypothetical keyword, would signature Greet(name: str) -> str(I hope this is not the wrong forum for hypothetical discussions like this) |
|
Note that such syntactical changes are hard to get accepted -- adding a reserved word to Python requires a PEP, a In contrast, a decorator is trivial to add -- you can just import it. |
|
I am fine with the current way. Some reasons:
class One(Protocol):
def __call__(self, x: T) -> T: ...
class Other(Protocol[T]):
def __call__(self, x: T) -> T: ...with the proposed syntax this would require supporting something like: @some_decorator(bind=(T,))
def Other(x: T) -> T: ...and add some tricky runtime machinery to allow |
|
I always sided with the line of thinking described by Guido here. I didn't even think we'd need a special decorator for it. I always thought of function objects as "types with callable signatures". I've seen many cases where being able to pass an example function as a type would make things clearer instead of using In practice I don't mind specifying a Protocol with |
|
It's a tough call. In the end I think we should just try to live with the Protocol solution for a while before we decide that it's too onerous. So I'm closing this. |
Right now you can specify callables with two patterns of arguments (shown here by example):
Callable[..., int]takes in any arguments, any number.Callable[[int, str, bool], int]takes in a predetermined number of required positional arguments, none of which have names specified.These don't cleanly match the actual types of callable objects in Python. Argument names, whether arguments are optional, and whether arguments are
*argsor**kwargsdo affect the type of a callable. We should be able to spell these things in the type language. Doing so would enable us to correctly write the types of callback functions, for example.Callable should take two arguments: an argument list and a return type. The return type is exactly as currently described in PEP484. The argument list is either:
..., indicating the function can take any arguments at all.An argument specifier is one of:
TYP. This has the same meaning asArg(TYP)Arg(type, name=None), indicating a positional argument. If the name is specified, the argument must have that name.OptionalArg(type, name=None), indicating an optional positional argument. If the name is specified, the argument must have that name. (alternate name possibilityOptArg(type, name=None)StarArg(type), indicating a "star argument" like*argsKwArg(type), indicating a "double star argument" like**kwargs. (an alternate name here would beStar2Arg(type).The round parens are an indication that these are not actual types but rather this new argument specifier thing.
Like the rules for python function arguments, all positional argspecs must come before all optional argspecs must come before zero or one star argspecs must come before zero or one kw argspecs.
This should be able to spell all function types you can make in python by defining single functions or methods, with the exception of functions that need
SelfTypeto be properly specified, which is an orthogonal concern.Some statements I think are true:
Callable[[Arg(T1, name='foo'), Arg(T2, name='bar')], R]is a subtype ofCallable[[T1, T2], R]Callable[[T1, OptionalArg(T2)], R]is a subtype ofCallable[[T1], R]Callable[[StarArg(T1)], R]is a subtype ofCallable[[], R]and is also a subtype ofCallable[[T1], R]and is also a subtype ofCallable[[T1, T1], R]and so on.Callable[[T1, StarArg(T1)], R]is a subtype ofCallable[[T1], R]and is also a subtype ofCallable[[T1, T1], R]and so on, but is not a subtype ofCallable[[], R]The text was updated successfully, but these errors were encountered: