-
-
Notifications
You must be signed in to change notification settings - Fork 30.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bpo-43428: Sync with importlib_metadata 3.7. #24782
Conversation
I added reviewers then realized tests are failing. I'll need to investigate why before requesting review. |
49a648b
to
b35f868
Compare
@rhettinger I didn't explicitly invite you. I think some bot did so on my behalf because this pull includes a new module with |
Looks like the docs build is failing due to an issue with the blurb. I'll work on that. |
Remaining test failure on macOS appears to be spurious, so I'm re-running GHA jobs. |
I want to get this change into the next alpha for feedback. Given the implementation has been released in importlib_metadata for weeks without reported incident, I feel confident about it. We can of course revert it if needed. I'd still welcome a review on it if anyone wishes to do so. |
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
External importlib repo doesn't have directives to indicate the Python version from which the function is available since it can be used with different versions of Python. This might cause the docs to have conflicts with repo but it will be good to have the relevant version in cpython docs.
distributions (in the case of a namespace package) for top-level | ||
Python packages or modules:: | ||
|
||
>>> packages_distributions() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will be good to document this as newly added in Python 3.10
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed in 690775a.
The failure seems to be in asyncio, so I'm disregarding. |
import functools | ||
import itertools | ||
import posixpath | ||
import collections | ||
import collections.abc |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
collections.abc
seems to be unused and is present only in docstring as collections.abc.Sequence
. collections.namedtuple
and collections.defaultdict
are used. I guess collections
is the correct import here and is imported as a side effect of collections.abc
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in python/importlib_metadata@4ae9584.
@@ -4,20 +4,24 @@ | |||
import csv | |||
import sys | |||
import email | |||
import inspect |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
inspect is used only for flake8 related utility and moving it to a local import can save little time. I might be benchmarking import time wrong with importtime so please verify my results.
./python -X importtime -c 'import importlib.metadata'
import time: self [us] | cumulative | imported package
import time: 366 | 366 | _io
import time: 89 | 89 | marshal
import time: 538 | 538 | posix
import time: 1212 | 2203 | _frozen_importlib_external
import time: 183 | 183 | time
import time: 544 | 726 | zipimport
import time: 107 | 107 | _codecs
import time: 1088 | 1194 | codecs
import time: 686 | 686 | encodings.aliases
import time: 1435 | 3313 | encodings
import time: 361 | 361 | encodings.utf_8
import time: 276 | 276 | _signal
import time: 78 | 78 | _abc
import time: 605 | 682 | abc
import time: 550 | 1231 | io
import time: 101 | 101 | _stat
import time: 580 | 681 | stat
import time: 2089 | 2089 | _collections_abc
import time: 295 | 295 | genericpath
import time: 584 | 878 | posixpath
import time: 1471 | 5117 | os
import time: 500 | 500 | _sitebuiltins
import time: 434 | 434 | sitecustomize
import time: 137 | 137 | usercustomize
import time: 1304 | 7490 | site
import time: 451 | 451 | warnings
import time: 558 | 1009 | importlib
import time: 443 | 443 | types
import time: 1834 | 2277 | enum
import time: 119 | 119 | _sre
import time: 513 | 513 | sre_constants
import time: 715 | 1228 | sre_parse
import time: 660 | 2005 | sre_compile
import time: 183 | 183 | itertools
import time: 279 | 279 | keyword
import time: 244 | 244 | _operator
import time: 602 | 846 | operator
import time: 343 | 343 | reprlib
import time: 123 | 123 | _collections
import time: 1716 | 3488 | collections
import time: 117 | 117 | _functools
import time: 1214 | 4818 | functools
import time: 145 | 145 | _locale
import time: 298 | 298 | copyreg
import time: 1864 | 11405 | re
import time: 367 | 367 | _csv
import time: 734 | 1100 | csv
import time: 292 | 292 | email
import time: 130 | 130 | _ast
import time: 939 | 939 | contextlib
import time: 2491 | 3559 | ast
import time: 264 | 264 | _opcode
import time: 635 | 898 | opcode
import time: 852 | 1749 | dis
import time: 385 | 385 | collections.abc
import time: 260 | 260 | importlib.machinery
import time: 312 | 312 | token
import time: 1924 | 2235 | tokenize
import time: 414 | 2649 | linecache
import time: 4521 | 4521 | typing
import time: 3599 | 16719 | inspect
import time: 424 | 424 | fnmatch
import time: 175 | 175 | nt
import time: 143 | 143 | nt
import time: 141 | 141 | nt
import time: 133 | 133 | nt
import time: 998 | 1589 | ntpath
import time: 132 | 132 | errno
import time: 282 | 282 | urllib
import time: 1822 | 2103 | urllib.parse
import time: 1827 | 6074 | pathlib
import time: 486 | 486 | binascii
import time: 271 | 271 | importlib._abc
import time: 1936 | 2206 | importlib.util
import time: 369 | 369 | zlib
import time: 449 | 449 | _compression
import time: 492 | 492 | _weakrefset
import time: 1096 | 1588 | threading
import time: 405 | 405 | _bz2
import time: 929 | 3369 | bz2
import time: 454 | 454 | _lzma
import time: 573 | 1027 | lzma
import time: 100 | 100 | pwd
import time: 330 | 330 | grp
import time: 1792 | 6984 | shutil
import time: 326 | 326 | _struct
import time: 372 | 697 | struct
import time: 1840 | 12211 | zipfile
import time: 244 | 244 | importlib._itertools
import time: 3696 | 3696 | configparser
import time: 951 | 951 | importlib.abc
import time: 13867 | 67563 | importlib.metadata
- Without
import inspect
./python -X importtime -c 'import importlib.metadata'
import time: self [us] | cumulative | imported package
import time: 362 | 362 | _io
import time: 72 | 72 | marshal
import time: 556 | 556 | posix
import time: 1321 | 2309 | _frozen_importlib_external
import time: 189 | 189 | time
import time: 482 | 670 | zipimport
import time: 95 | 95 | _codecs
import time: 1011 | 1105 | codecs
import time: 633 | 633 | encodings.aliases
import time: 1187 | 2924 | encodings
import time: 350 | 350 | encodings.utf_8
import time: 269 | 269 | _signal
import time: 79 | 79 | _abc
import time: 572 | 651 | abc
import time: 631 | 1281 | io
import time: 95 | 95 | _stat
import time: 538 | 633 | stat
import time: 1869 | 1869 | _collections_abc
import time: 255 | 255 | genericpath
import time: 473 | 727 | posixpath
import time: 1181 | 4408 | os
import time: 324 | 324 | _sitebuiltins
import time: 408 | 408 | sitecustomize
import time: 141 | 141 | usercustomize
import time: 1182 | 6461 | site
import time: 555 | 555 | warnings
import time: 560 | 1114 | importlib
import time: 478 | 478 | types
import time: 1852 | 2330 | enum
import time: 121 | 121 | _sre
import time: 515 | 515 | sre_constants
import time: 1579 | 2094 | sre_parse
import time: 724 | 2938 | sre_compile
import time: 237 | 237 | itertools
import time: 519 | 519 | keyword
import time: 217 | 217 | _operator
import time: 1186 | 1403 | operator
import time: 345 | 345 | reprlib
import time: 125 | 125 | _collections
import time: 2238 | 4864 | collections
import time: 109 | 109 | _functools
import time: 1157 | 6129 | functools
import time: 180 | 180 | _locale
import time: 378 | 378 | copyreg
import time: 1596 | 13549 | re
import time: 408 | 408 | _csv
import time: 912 | 1319 | csv
import time: 313 | 313 | email
import time: 289 | 289 | fnmatch
import time: 233 | 233 | nt
import time: 150 | 150 | nt
import time: 139 | 139 | nt
import time: 135 | 135 | nt
import time: 777 | 1431 | ntpath
import time: 121 | 121 | errno
import time: 276 | 276 | urllib
import time: 2085 | 2360 | urllib.parse
import time: 1608 | 5808 | pathlib
import time: 516 | 516 | binascii
import time: 254 | 254 | importlib._abc
import time: 1029 | 1029 | contextlib
import time: 677 | 1958 | importlib.util
import time: 351 | 351 | zlib
import time: 410 | 410 | _compression
import time: 655 | 655 | _weakrefset
import time: 1484 | 2138 | threading
import time: 406 | 406 | _bz2
import time: 722 | 3675 | bz2
import time: 484 | 484 | _lzma
import time: 557 | 1040 | lzma
import time: 108 | 108 | pwd
import time: 384 | 384 | grp
import time: 1506 | 7062 | shutil
import time: 360 | 360 | _struct
import time: 408 | 767 | struct
import time: 1973 | 12275 | zipfile
import time: 242 | 242 | importlib._itertools
import time: 561 | 561 | collections.abc
import time: 4142 | 4703 | configparser
import time: 246 | 246 | importlib.machinery
import time: 2303 | 2303 | _ast
import time: 2311 | 4614 | ast
import time: 4474 | 9087 | typing
import time: 1303 | 10635 | importlib.abc
import time: 3051 | 53004 | importlib.metadata
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in python/importlib_metadata@15312a7.
@@ -617,16 +765,28 @@ def version(distribution_name): | |||
return distribution(distribution_name).version | |||
|
|||
|
|||
def entry_points(): | |||
def entry_points(**params) -> Union[EntryPoints, SelectableGroups]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This represents a change in signature for the public function that selectors can be passed directly like below which is not possible before the update. There are no separate docs for entry_points
as a function but it will be good to document this in the rst docs since docstring is updated but the rst docs still show entry_points()
not accepting parameters.
./python -Wall
Python 3.10.0a6+ (heads/master:f917efccf8, Mar 13 2021, 17:24:55) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.metadata as me
>>> me.entry_points(name="eggsecutable")
(EntryPoint(name='eggsecutable', value='setuptools.command.easy_install:bootstrap', group='setuptools.installation'),)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In python/importlib_metadata@bb24370, I expanded the docs to address this concern. It should be much clearer now what versions support the selection interface. Please take a look and let me know if that doesn't fully address the concern.
""" | ||
msg = ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The deprecation might be noted in docs since this was a valid usage before. The deprecation message is slightly unclear to me since EntryPoints
object is not documented.
./python -Wall
Python 3.10.0a6+ (heads/master:f917efccf8, Mar 13 2021, 17:24:55) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.metadata as me
>>> dict(me.entry_points()['setuptools.installation'])
/root/cpython/Lib/importlib/metadata.py:139: DeprecationWarning: Construction of dict of EntryPoints is deprecated in favor of EntryPoints.
warnings.warn(msg, DeprecationWarning)
{'eggsecutable': EntryPoint(name='eggsecutable', value='setuptools.command.easy_install:bootstrap', group='setuptools.installation')}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This deprecation is unrelated to the introduction of EntryPoints
or the selection interface. You can get the same warning with:
>>> dict(me.entry_points(group='setuptools.installation'))
Previously, an EntryPoint
object allowed itself to be used as a name/value pair during construction of a dictionary from a series of EntryPoint
objects. This behavior was somewhat obtuse, so I don't expect it to be in widespread use.
Oh, I see. It's unclear from a user's perspective what EntryPoints
is. In python/importlib_metadata@bb24370, they are now documented.
Additionally, in python/importlib_metadata@bf777ae, I've added documentation for EntryPoints
.
What's new document also has a separate section on improved modules where public function like |
What's new document also has a separate section on improved modules where public function like |
|
||
from configparser import ConfigParser | ||
from contextlib import suppress | ||
from importlib import import_module | ||
from importlib.abc import MetaPathFinder | ||
from itertools import starmap | ||
from typing import Any, List, Optional, Protocol, TypeVar, Union | ||
from typing import Any, List, Mapping, Optional, Protocol, TypeVar, Union | ||
|
||
|
||
__all__ = [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can be updated with packages_distributions
as a public function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added to python/importlib_metadata@7bdeaa4, to be included in a future sync.
…o version 3.7.3 v3.7.3 ====== * Docs enhancements and cleanup following review in `GH-24782 <https://github.com/python/cpython/pull/24782>`_.
v4.10.1 ======= * #361: Avoid potential REDoS in ``EntryPoint.pattern``. v4.10.0 ======= * #354: Removed ``Distribution._local`` factory. This functionality was created as a demonstration of the possible implementation. Now, the `pep517 <https://pypi.org/project/pep517>`_ package provides this functionality directly through `pep517.meta.load <https://github.com/pypa/pep517/blob/a942316305395f8f757f210e2b16f738af73f8b8/pep517/meta.py#L63-L73>`_. v4.9.0 ====== * Require Python 3.7 or later. v4.8.3 ====== * #357: Fixed requirement generation from egg-info when a URL requirement is given. v4.8.2 ====== v2.1.2 ====== * #353: Fixed discovery of distributions when path is empty. v4.8.1 ====== * #348: Restored support for ``EntryPoint`` access by item, deprecating support in the process. Users are advised to use direct member access instead of item-based access:: - ep[0] -> ep.name - ep[1] -> ep.value - ep[2] -> ep.group - ep[:] -> ep.name, ep.value, ep.group v4.8.0 ====== * #337: Rewrote ``EntryPoint`` as a simple class, still immutable and still with the attributes, but without any expectation for ``namedtuple`` functionality such as ``_asdict``. v4.7.1 ====== * #344: Fixed regression in ``packages_distributions`` when neither top-level.txt nor a files manifest is present. v4.7.0 ====== * #330: In ``packages_distributions``, now infer top-level names from ``.files()`` when a ``top-level.txt`` (Setuptools-specific metadata) is not present. v4.6.4 ====== * #334: Correct ``SimplePath`` protocol to match ``pathlib`` protocol for ``__truediv__``. v4.6.3 ====== * Moved workaround for #327 to ``_compat`` module. v4.6.2 ====== * bpo-44784: Avoid errors in test suite when DeprecationWarnings are treated as errors. v4.6.1 ====== * #327: Deprecation warnings now honor call stack variance on PyPy. v4.6.0 ====== * #326: Performance tests now rely on `pytest-perf <https://pypi.org/project/pytest-perf>`_. To disable these tests, which require network access and a git checkout, pass ``-p no:perf`` to pytest. v4.5.0 ====== * #319: Remove ``SelectableGroups`` deprecation exception for flake8. v4.4.0 ====== * #300: Restore compatibility in the result from ``Distribution.entry_points`` (``EntryPoints``) to honor expectations in older implementations and issuing deprecation warnings for these cases: - ``EntryPoints`` objects are once again mutable, allowing for ``sort()`` and other list-based mutation operations. Avoid deprecation warnings by casting to a mutable sequence (e.g. ``list(dist.entry_points).sort()``). - ``EntryPoints`` results once again allow for access by index. To avoid deprecation warnings, cast the result to a Sequence first (e.g. ``tuple(dist.entry_points)[0]``). v4.3.1 ====== * #320: Fix issue where normalized name for eggs was incorrectly solicited, leading to metadata being unavailable for eggs. v4.3.0 ====== * #317: De-duplication of distributions no longer requires loading the full metadata for ``PathDistribution`` objects, entry point loading performance by ~10x. v4.2.0 ====== * Prefer f-strings to ``.format`` calls. v4.1.0 ====== * #312: Add support for metadata 2.2 (``Dynamic`` field). * #315: Add ``SimplePath`` protocol for interface clarity in ``PathDistribution``. v4.0.1 ====== * #306: Clearer guidance about compatibility in readme. v4.0.0 ====== * #304: ``PackageMetadata`` as returned by ``metadata()`` and ``Distribution.metadata()`` now provides normalized metadata honoring PEP 566: - If a long description is provided in the payload of the RFC 822 value, it can be retrieved as the ``Description`` field. - Any multi-line values in the metadata will be returned as such. - For any multi-line values, line continuation characters are removed. This backward-incompatible change means that any projects relying on the RFC 822 line continuation characters being present must be tolerant to them having been removed. - Add a ``json`` property that provides the metadata converted to a JSON-compatible form per PEP 566. v3.10.1 ======= * Minor tweaks from CPython. v3.10.0 ======= * #295: Internal refactoring to unify section parsing logic. v3.9.1 ====== * #296: Exclude 'prepare' package. * #297: Fix ValueError when entry points contains comments. v3.9.0 ====== * Use of Mapping (dict) interfaces on ``SelectableGroups`` is now flagged as deprecated. Instead, users are advised to use the select interface for future compatibility. Suppress the warning with this filter: ``ignore:SelectableGroups dict interface``. Or with this invocation in the Python environment: ``warnings.filterwarnings('ignore', 'SelectableGroups dict interface')``. Preferably, switch to the ``select`` interface introduced in 3.7.0. See the `entry points documentation <https://importlib-metadata.readthedocs.io/en/latest/using.html#entry-points>`_ and changelog for the 3.6 release below for more detail. For some use-cases, especially those that rely on ``importlib.metadata`` in Python 3.8 and 3.9 or those relying on older ``importlib_metadata`` (especially on Python 3.5 and earlier), `backports.entry_points_selectable <https://pypi.org/project/backports.entry_points_selectable>`_ was created to ease the transition. Please have a look at that project if simply relying on importlib_metadata 3.6+ is not straightforward. Background in #298. * #283: Entry point parsing no longer relies on ConfigParser and instead uses a custom, one-pass parser to load the config, resulting in a ~20% performance improvement when loading entry points. v3.8.2 ====== * #293: Re-enabled lazy evaluation of path lookup through a FreezableDefaultDict. v3.8.1 ====== * #293: Workaround for error in distribution search. v3.8.0 ====== * #290: Add mtime-based caching for ``FastPath`` and its lookups, dramatically increasing performance for repeated distribution lookups. v3.7.3 ====== * Docs enhancements and cleanup following review in `GH-24782 <https://github.com/python/cpython/pull/24782>`_. v3.7.2 ====== * Cleaned up cruft in entry_points docstring. v3.7.1 ====== * Internal refactoring to facilitate ``entry_points() -> dict`` deprecation. v3.7.0 ====== * #131: Added ``packages_distributions`` to conveniently resolve a top-level package or module to its distribution(s). v3.6.0 ====== * #284: Introduces new ``EntryPoints`` object, a tuple of ``EntryPoint`` objects but with convenience properties for selecting and inspecting the results: - ``.select()`` accepts ``group`` or ``name`` keyword parameters and returns a new ``EntryPoints`` tuple with only those that match the selection. - ``.groups`` property presents all of the group names. - ``.names`` property presents the names of the entry points. - Item access (e.g. ``eps[name]``) retrieves a single entry point by name. ``entry_points`` now accepts "selection parameters", same as ``EntryPoint.select()``. ``entry_points()`` now provides a future-compatible ``SelectableGroups`` object that supplies the above interface (except item access) but remains a dict for compatibility. In the future, ``entry_points()`` will return an ``EntryPoints`` object for all entry points. If passing selection parameters to ``entry_points``, the future behavior is invoked and an ``EntryPoints`` is the result. * #284: Construction of entry points using ``dict([EntryPoint, ...])`` is now deprecated and raises an appropriate DeprecationWarning and will be removed in a future version. * #300: ``Distribution.entry_points`` now presents as an ``EntryPoints`` object and access by index is no longer allowed. If access by index is required, cast to a sequence first. v3.5.0 ====== * #280: ``entry_points`` now only returns entry points for unique distributions (by name). v3.4.0 ====== * #10: Project now declares itself as being typed. * #272: Additional performance enhancements to distribution discovery. * #111: For PyPA projects, add test ensuring that ``MetadataPathFinder._search_paths`` honors the needed interface. Method is still private. v3.3.0 ====== * #265: ``EntryPoint`` objects now expose a ``.dist`` object referencing the ``Distribution`` when constructed from a Distribution. v3.2.0 ====== * The object returned by ``metadata()`` now has a formally-defined protocol called ``PackageMetadata`` with declared support for the ``.get_all()`` method. Fixes #126. v3.1.1 ====== v2.1.1 ====== * #261: Restored compatibility for package discovery for metadata without version in the name and for legacy eggs. v3.1.0 ====== * Merge with 2.1.0. v2.1.0 ====== * #253: When querying for package metadata, the lookup now honors `package normalization rules <https://packaging.python.org/specifications/recording-installed-packages/>`_. v3.0.0 ====== * Require Python 3.6 or later.
@jaraco, in python/miss-islington#590 we are seeing a failure that appears to be related to this PR. Can you look into it and confirm whether it's related or not? |
https://bugs.python.org/issue43428
Automerge-Triggered-By: GH:jaraco