-
-
Notifications
You must be signed in to change notification settings - Fork 30.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gh-109653: typing.py
: improve import time by creating soft-deprecated members on demand
#109651
Conversation
🤖 New build scheduled with the buildbot fleet by @AlexWaygood for commit 180461f 🤖 If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again. |
typing.py
typing.py
typing.py
typing.py
: improve import time by creating soft-deprecated members on demand
Would it be going to far to use this approach for all of the If it would be, perhaps we could defer the A |
The main reason why the changes here save us so much is because we can avoid unconditionally importing
IIRC, |
I can just about see a path to inlining A |
FWIW @AA-Turner, here's a diff (relative to this PR branch) that shaves another 0.002s off the import time by avoiding the diff --git a/Lib/typing.py b/Lib/typing.py
index 84b741bfb0..0218b715f6 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -19,9 +19,7 @@
"""
from abc import abstractmethod, ABCMeta
-import collections
-from collections import defaultdict
-import collections.abc
+import _collections_abc
import copyreg
import functools
import operator
@@ -40,6 +38,11 @@
Generic,
)
+try:
+ from _collections import defaultdict
+except ImportError:
+ from collections import defaultdict
+
# Please keep __all__ alphabetized within each category.
__all__ = [
# Super-special typing primitives.
@@ -219,7 +222,7 @@ def _should_unflatten_callable_args(typ, args):
we need to unflatten it.
"""
return (
- typ.__origin__ is collections.abc.Callable
+ typ.__origin__ is _collections_abc.Callable
and not (len(args) == 2 and _is_param_expr(args[0]))
)
@@ -1317,7 +1320,7 @@ def _make_substitution(self, args, new_arg_by_param):
subargs.append(new_arg_by_param[x])
new_arg = old_arg[tuple(subargs)]
- if self.__origin__ == collections.abc.Callable and isinstance(new_arg, tuple):
+ if self.__origin__ == _collections_abc.Callable and isinstance(new_arg, tuple):
# Consider the following `Callable`.
# C = Callable[[int], str]
# Here, `C.__args__` should be (int, str) - NOT ([int], str).
@@ -1885,7 +1888,7 @@ def _proto_hook(cls, other):
# ...or in annotations, if it is a sub-protocol.
annotations = getattr(base, '__annotations__', {})
- if (isinstance(annotations, collections.abc.Mapping) and
+ if (isinstance(annotations, _collections_abc.Mapping) and
attr in annotations and
issubclass(other, Generic) and getattr(other, '_is_protocol', False)):
break
@@ -2521,18 +2524,18 @@ class Other(Leaf): # Error reported by type checker
# Various ABCs mimicking those in collections.abc.
_alias = _SpecialGenericAlias
-Hashable = _alias(collections.abc.Hashable, 0) # Not generic.
-Awaitable = _alias(collections.abc.Awaitable, 1)
-Coroutine = _alias(collections.abc.Coroutine, 3)
-AsyncIterable = _alias(collections.abc.AsyncIterable, 1)
-AsyncIterator = _alias(collections.abc.AsyncIterator, 1)
-Iterable = _alias(collections.abc.Iterable, 1)
-Iterator = _alias(collections.abc.Iterator, 1)
-Reversible = _alias(collections.abc.Reversible, 1)
-Sized = _alias(collections.abc.Sized, 0) # Not generic.
-Container = _alias(collections.abc.Container, 1)
-Collection = _alias(collections.abc.Collection, 1)
-Callable = _CallableType(collections.abc.Callable, 2)
+Hashable = _alias(_collections_abc.Hashable, 0) # Not generic.
+Awaitable = _alias(_collections_abc.Awaitable, 1)
+Coroutine = _alias(_collections_abc.Coroutine, 3)
+AsyncIterable = _alias(_collections_abc.AsyncIterable, 1)
+AsyncIterator = _alias(_collections_abc.AsyncIterator, 1)
+Iterable = _alias(_collections_abc.Iterable, 1)
+Iterator = _alias(_collections_abc.Iterator, 1)
+Reversible = _alias(_collections_abc.Reversible, 1)
+Sized = _alias(_collections_abc.Sized, 0) # Not generic.
+Container = _alias(_collections_abc.Container, 1)
+Collection = _alias(_collections_abc.Collection, 1)
+Callable = _CallableType(_collections_abc.Callable, 2)
Callable.__doc__ = \
"""Deprecated alias to collections.abc.Callable.
@@ -2547,15 +2550,15 @@ class Other(Leaf): # Error reported by type checker
There is no syntax to indicate optional or keyword arguments;
such function types are rarely used as callback types.
"""
-AbstractSet = _alias(collections.abc.Set, 1, name='AbstractSet')
-MutableSet = _alias(collections.abc.MutableSet, 1)
+AbstractSet = _alias(_collections_abc.Set, 1, name='AbstractSet')
+MutableSet = _alias(_collections_abc.MutableSet, 1)
# NOTE: Mapping is only covariant in the value type.
-Mapping = _alias(collections.abc.Mapping, 2)
-MutableMapping = _alias(collections.abc.MutableMapping, 2)
-Sequence = _alias(collections.abc.Sequence, 1)
-MutableSequence = _alias(collections.abc.MutableSequence, 1)
+Mapping = _alias(_collections_abc.Mapping, 2)
+MutableMapping = _alias(_collections_abc.MutableMapping, 2)
+Sequence = _alias(_collections_abc.Sequence, 1)
+MutableSequence = _alias(_collections_abc.MutableSequence, 1)
ByteString = _DeprecatedGenericAlias(
- collections.abc.ByteString, 0, removal_version=(3, 14) # Not generic.
+ _collections_abc.ByteString, 0, removal_version=(3, 14) # Not generic.
)
# Tuple accepts variable number of parameters.
Tuple = _TupleType(tuple, -1, inst=False, name='Tuple')
@@ -2571,20 +2574,15 @@ class Other(Leaf): # Error reported by type checker
To specify a variable-length tuple of homogeneous type, use Tuple[T, ...].
"""
List = _alias(list, 1, inst=False, name='List')
-Deque = _alias(collections.deque, 1, name='Deque')
Set = _alias(set, 1, inst=False, name='Set')
FrozenSet = _alias(frozenset, 1, inst=False, name='FrozenSet')
-MappingView = _alias(collections.abc.MappingView, 1)
-KeysView = _alias(collections.abc.KeysView, 1)
-ItemsView = _alias(collections.abc.ItemsView, 2)
-ValuesView = _alias(collections.abc.ValuesView, 1)
+MappingView = _alias(_collections_abc.MappingView, 1)
+KeysView = _alias(_collections_abc.KeysView, 1)
+ItemsView = _alias(_collections_abc.ItemsView, 2)
+ValuesView = _alias(_collections_abc.ValuesView, 1)
Dict = _alias(dict, 2, inst=False, name='Dict')
-DefaultDict = _alias(collections.defaultdict, 2, name='DefaultDict')
-OrderedDict = _alias(collections.OrderedDict, 2)
-Counter = _alias(collections.Counter, 1)
-ChainMap = _alias(collections.ChainMap, 2)
-Generator = _alias(collections.abc.Generator, 3)
-AsyncGenerator = _alias(collections.abc.AsyncGenerator, 2)
+Generator = _alias(_collections_abc.Generator, 3)
+AsyncGenerator = _alias(_collections_abc.AsyncGenerator, 2)
Type = _alias(type, 1, inst=False, name='Type')
Type.__doc__ = \
"""Deprecated alias to builtins.type.
@@ -2692,8 +2690,8 @@ def _make_nmtuple(name, types, module, defaults = ()):
fields = [n for n, t in types]
types = {n: _type_check(t, f"field {n} annotation must be a type")
for n, t in types}
- nm_tpl = collections.namedtuple(name, fields,
- defaults=defaults, module=module)
+ from collections import namedtuple
+ nm_tpl = namedtuple(name, fields, defaults=defaults, module=module)
nm_tpl.__annotations__ = nm_tpl.__new__.__annotations__ = types
return nm_tpl
@@ -3432,6 +3430,17 @@ def __getattr__(attr):
elif attr in {"ContextManager", "AsyncContextManager"}:
import contextlib
obj = _alias(getattr(contextlib, f"Abstract{attr}"), 1, name=attr)
+ elif attr in {"Deque", "DefaultDict", "OrderedDict", "Counter", "ChainMap"}:
+ import collections
+ match attr:
+ case "Deque":
+ obj = _alias(collections.deque, 1, name="Deque")
+ case "DefaultDict":
+ obj = _alias(collections.defaultdict, 2, name="DefaultDict")
+ case "OrderedDict" | "ChainMap":
+ obj = _alias(getattr(collections, attr), 2)
+ case "Counter":
+ obj = _alias(collections.Counter, 1)
else:
raise AttributeError(f"Module 'typing' has no attribute {attr!r}")
globals()[attr] = obj |
Though still 25% (on the 8ms post-this PR) or 50% total incl. this PR. I agree this PR is both bigger bang-for-buck and should come first (I've no comments save applying Tom's suggestion), but for e.g. CLIs, milli-seconds can matter in feeling responsive, so I think worth at least considering the further improvement! A |
I'm a bit skeptical of this because it relies on avoiding imports of common standard library modules only: how many real applications are there going to be that want to import On the other hand, the change in this PR is fairly small and self-contained, so I won't oppose it. But we shouldn't go through extremely lengths to avoid importing common stdlib modules. |
Co-authored-by: Thomas Grainger <[email protected]>
Well, for one example, from a quick grep, More than that, though, I think that it's generally good to do this kind of thing (where it's not significantly detrimental to code readability). It's best to reduce dependencies between stdlib modules wherever possible, and make it clearer exactly why certain stdlib modules depend on others. It helps reduce the extent to which the stdlib is a massive import cycle.
I definitely agree with you there :) |
But they do use |
Sure, it's unlikely that this PR is going to hugely improve the import time of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Those are good points. Let's do it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! I'm fine with this, but #109651 (comment) crosses the line for me.
Thanks all for the helpful reviews! |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
…precated members on demand (python#109651) Co-authored-by: Thomas Grainger <[email protected]>
…precated members on demand (python#109651) Co-authored-by: Thomas Grainger <[email protected]>
This cuts about 1/3 off the import time of
typing.py
for me locally (PGO-optimised, non-debug build). 0.012s -> 0.008s.Benchmark script
(There's probably a better way of benchmarking import times, but this seems to work pretty well.)