Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve import time of various stdlib modules #118761

Open
layday opened this issue May 8, 2024 · 5 comments
Open

Improve import time of various stdlib modules #118761

layday opened this issue May 8, 2024 · 5 comments
Labels
performance Performance or resource usage topic-importlib type-feature A feature request or enhancement

Comments

@layday
Copy link

layday commented May 8, 2024

Feature or enhancement

Proposal:

Following on from #109653, further improvements can be made to import times.

Links to previous discussion of this feature:

https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173

For example:

importlib.metadata is often used for tasks that need to happen at import, e.g. to enumerate/load entry point plug-ins, so it might be worth seeing if we can cut down its own import time a bit more.

importlib.metadata imports zipfile at the top for a function that won't be called in the vast majority of cases. It also imports importlib.abc, which in turn imports importlib.resources, to subclass an ABC with a single, non-abstract method - I assume redefining the method in importlib.metadata would be harmless. Some other less frequently-used imports which are only accessed once or twice, such as json, could also be tucked away in their calling functions.

Linked PRs

@AlexWaygood AlexWaygood added topic-importlib performance Performance or resource usage type-feature A feature request or enhancement labels May 8, 2024
@hugovk
Copy link
Member

hugovk commented Aug 5, 2024

@layday Is it okay if I repurpose this issue as an "Improve import time of various stdlib modules" like #109653 but for 3.14?

I've got some pprint improvements, and if we have importlib.metadata and some others, we can group them under the same umbrella issue like last time.

@layday
Copy link
Author

layday commented Aug 5, 2024

Sure!

@danielhollas
Copy link
Contributor

danielhollas commented Aug 6, 2024

I've opened a PR over at the importlib_metadata repo that avoids importing inspect. python/importlib_metadata#499

importlib.metadata imports zipfile at the top for a function that won't be called in the vast majority of cases.
Some other less frequently-used imports which are only accessed once or twice, such as json, could also be tucked away in their calling functions.

@layday were you planning on tackling these?

It also imports importlib.abc, which in turn imports importlib.resources, to subclass an ABC with a single, non-abstract method

This seems to be solved on main, importlib.abc no longer imports importlib.resources.

@hugovk hugovk changed the title Further improve import time of importlib.metadata Improve import time of various stdlib modules Aug 6, 2024
@hugovk hugovk added the 3.14 new features, bugs and security fixes label Aug 6, 2024
hugovk added a commit that referenced this issue Aug 7, 2024
blhsing pushed a commit to blhsing/cpython that referenced this issue Aug 22, 2024
@danielhollas
Copy link
Contributor

I've opened a PR over at the importlib_metadata repo that avoids importing inspect. python/importlib_metadata#499

This has been merged and released in version 8.4 of importlib_metadata 🎉

importlib.metadata imports zipfile at the top for a function that won't be called in the vast majority of cases. It also imports importlib.abc, which in turn imports importlib.resources, to subclass an ABC with a single, non-abstract method - I assume redefining the method in importlib.metadata would be harmless. Some other less frequently-used imports which are only accessed once or twice, such as json, could also be tucked away in their calling functions.

I've submitted python/importlib_metadata#502 that defers zip import, and python/importlib_metadata#503 which defers json and platform.

@picnixz picnixz removed the 3.14 new features, bugs and security fixes label Aug 31, 2024
@picnixz
Copy link
Contributor

picnixz commented Aug 31, 2024

(removing the 3.14 label since features always target the main branch)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Performance or resource usage topic-importlib type-feature A feature request or enhancement
Projects
None yet
Development

No branches or pull requests

5 participants