#339 / v4.8.0 broke stevedore which broke bandit, openstackclient, flake8 and others #348

mr-c · 2021-08-29T10:08:52Z

Hello, #339 in v4.8.0 broke stevedore which broke bandit for at least myself (but probably others)

https://github.com/common-workflow-language/cwltool/pull/1482/checks?check_run_id=3454232416#step:9:50
PyCQA/bandit#730

Reverting to importlib_metadata version 4.7.1 resolves the problem for me

Originally posted by @mr-c in #339 (comment)

The text was updated successfully, but these errors were encountered:

Stannislav · 2021-08-29T10:13:51Z

Same

$ bandit
Traceback (most recent call last):
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/stevedore/_cache.py", line 159, in _get_data_for_path
    with open(filename, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/me/Library/Caches/Python Entry Points/f02c0770d01da06fbfcfe6435c8feb2e12f0e5fb7e239f6cf91d4d8d088821d8'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/me/project/.tox/lint/bin/bandit", line 5, in <module>
    from bandit.cli.main import main
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/bandit/__init__.py", line 19, in <module>
    from bandit.core import config  # noqa
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/bandit/core/__init__.py", line 17, in <module>
    from bandit.core import config  # noqa
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/bandit/core/config.py", line 12, in <module>
    from bandit.core import extension_loader
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/bandit/core/extension_loader.py", line 109, in <module>
    MANAGER = Manager()
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/bandit/core/extension_loader.py", line 25, in __init__
    self.load_formatters(formatters_namespace)
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/bandit/core/extension_loader.py", line 33, in load_formatters
    verify_requirements=False,
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/stevedore/extension.py", line 136, in __init__
    verify_requirements)
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/stevedore/extension.py", line 218, in _load_plugins
    for ep in self.list_entry_points():
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/stevedore/extension.py", line 207, in list_entry_points
    eps = list(_cache.get_group_all(self.namespace))
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/stevedore/_cache.py", line 179, in get_group_all
    data = self._get_data_for_path(path)
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/stevedore/_cache.py", line 162, in _get_data_for_path
    data = _build_cacheable_data(path)
  File "/Users/me/project/.tox/lint/lib/python3.7/site-packages/stevedore/_cache.py", line 119, in _build_cacheable_data
    item = ep[:]  # convert namedtuple to tuple
TypeError: 'EntryPoint' object is not subscriptable

freedge · 2021-08-29T11:45:17Z

Seems to break openstack cli and rally testing as well. Opening https://storyboard.openstack.org/#!/story/2009151

WilliamDEdwards · 2021-08-29T12:44:43Z

Same here with flake8 (uses stevedore). I created a stevedore bug here: https://bugs.launchpad.net/python-stevedore/+bug/1941991

jaraco · 2021-08-29T13:29:57Z

Sorry for the inconvenience. I've yanked 4.8.0 while working out a remedy.

The recent release of importlib-metadata has broken an interface that stevedore uses when looking for entrypoints (see python/importlib_metadata#348 ). Several of our test/ci dependecies use stevedore for their plugin interfaces including stestr which is causing CI failures. To unblock CI this commit pins the importlib metadata version in our constraints file while the upstream issue is resolved.

The recent release of importlib-metadata has broken an interface that stevedore uses when looking for entrypoints (see: python/importlib_metadata#348 ). Several of our test/ci dependecies use stevedore for their plugin interfaces including stestr which is causing CI failures. To unblock CI this commit pins the importlib metadata version in our constraints file while the upstream issue is resolved.

jaraco · 2021-08-29T13:36:58Z

I'd like devise a way to capture these requirements. I can see from the failure that stevedore is expecting __getitem__(slice) to return a tuple, something that namedtuple (or other tuple) would do implicitly.

jaraco · 2021-08-29T13:38:28Z

@mtreinish: You may wish to refrain from pinning too many projects as I've yanked the offending version and expect to release the next version without the broken behavior.

mr-c · 2021-08-29T13:46:13Z

Sorry for the inconvenience. I've yanked 4.8.0 while working out a remedy.

Thanks for the quick yank! I know this isn't nice news to receive.

jaraco · 2021-08-29T14:07:47Z

In #349, I've drafted a fix but marking the access by item as deprecated. I can add other missing but expected tuple behaviors there as well if needed. I welcome feedback.

jaraco · 2021-08-29T15:14:27Z

v4.8.1 is releasing now. I believe this addresses the issue. It's possible there are other usages out there dependent on tuple behaviors. If so, please raise an issue, mention me, and I'll address those promptly.

v4.8.1 #348: Restored support for EntryPoint access by item, deprecating support in the process. Users are advised to use direct member access instead of item-based access: - ep[0] -> ep.name - ep[1] -> ep.value - ep[2] -> ep.group - ep[:] -> ep.name, ep.value, ep.group v4.8.0 #337: Rewrote EntryPoint as a simple class, still immutable and still with the attributes, but without any expectation for namedtuple functionality such as _asdict. v4.7.1 #344: Fixed regression in packages_distributions when neither top-level.txt nor a files manifest is present. v4.7.0 #330: In packages_distributions, now infer top-level names from .files() when a top-level.txt (Setuptools-specific metadata) is not present. References: python/importlib_metadata#348 python/importlib_metadata#337 python/importlib_metadata#344 python/importlib_metadata#330 Signed-off-by: Tim Orling <[email protected]> Signed-off-by: Richard Purdie <[email protected]>

v4.8.1 #348: Restored support for EntryPoint access by item, deprecating support in the process. Users are advised to use direct member access instead of item-based access: - ep[0] -> ep.name - ep[1] -> ep.value - ep[2] -> ep.group - ep[:] -> ep.name, ep.value, ep.group v4.8.0 #337: Rewrote EntryPoint as a simple class, still immutable and still with the attributes, but without any expectation for namedtuple functionality such as _asdict. v4.7.1 #344: Fixed regression in packages_distributions when neither top-level.txt nor a files manifest is present. v4.7.0 #330: In packages_distributions, now infer top-level names from .files() when a top-level.txt (Setuptools-specific metadata) is not present. References: python/importlib_metadata#348 python/importlib_metadata#337 python/importlib_metadata#344 python/importlib_metadata#330 (From OE-Core rev: 01eb9d4384ae78b02780cea3b8690d99484b2602) Signed-off-by: Tim Orling <[email protected]> Signed-off-by: Richard Purdie <[email protected]>

v4.8.1 #348: Restored support for EntryPoint access by item, deprecating support in the process. Users are advised to use direct member access instead of item-based access: - ep[0] -> ep.name - ep[1] -> ep.value - ep[2] -> ep.group - ep[:] -> ep.name, ep.value, ep.group v4.8.0 #337: Rewrote EntryPoint as a simple class, still immutable and still with the attributes, but without any expectation for namedtuple functionality such as _asdict. v4.7.1 #344: Fixed regression in packages_distributions when neither top-level.txt nor a files manifest is present. v4.7.0 #330: In packages_distributions, now infer top-level names from .files() when a top-level.txt (Setuptools-specific metadata) is not present. References: python/importlib_metadata#348 python/importlib_metadata#337 python/importlib_metadata#344 python/importlib_metadata#330 (From OE-Core rev: 4272ca45d137b91ec368c94b3e0dbd7d56c616dd) Signed-off-by: Tim Orling <[email protected]> Signed-off-by: Richard Purdie <[email protected]>

v4.8.1 #348: Restored support for EntryPoint access by item, deprecating support in the process. Users are advised to use direct member access instead of item-based access: - ep[0] -> ep.name - ep[1] -> ep.value - ep[2] -> ep.group - ep[:] -> ep.name, ep.value, ep.group v4.8.0 #337: Rewrote EntryPoint as a simple class, still immutable and still with the attributes, but without any expectation for namedtuple functionality such as _asdict. v4.7.1 #344: Fixed regression in packages_distributions when neither top-level.txt nor a files manifest is present. v4.7.0 #330: In packages_distributions, now infer top-level names from .files() when a top-level.txt (Setuptools-specific metadata) is not present. References: python/importlib_metadata#348 python/importlib_metadata#337 python/importlib_metadata#344 python/importlib_metadata#330 Signed-off-by: Tim Orling <[email protected]> Signed-off-by: Richard Purdie <[email protected]>

v4.8.1 #348: Restored support for EntryPoint access by item, deprecating support in the process. Users are advised to use direct member access instead of item-based access: - ep[0] -> ep.name - ep[1] -> ep.value - ep[2] -> ep.group - ep[:] -> ep.name, ep.value, ep.group v4.8.0 #337: Rewrote EntryPoint as a simple class, still immutable and still with the attributes, but without any expectation for namedtuple functionality such as _asdict. v4.7.1 #344: Fixed regression in packages_distributions when neither top-level.txt nor a files manifest is present. v4.7.0 #330: In packages_distributions, now infer top-level names from .files() when a top-level.txt (Setuptools-specific metadata) is not present. References: python/importlib_metadata#348 python/importlib_metadata#337 python/importlib_metadata#344 python/importlib_metadata#330 Signed-off-by: Tim Orling <[email protected]> Signed-off-by: Alexandre Belloni <[email protected]> Signed-off-by: Richard Purdie <[email protected]>

v4.8.1 #348: Restored support for EntryPoint access by item, deprecating support in the process. Users are advised to use direct member access instead of item-based access: - ep[0] -> ep.name - ep[1] -> ep.value - ep[2] -> ep.group - ep[:] -> ep.name, ep.value, ep.group v4.8.0 #337: Rewrote EntryPoint as a simple class, still immutable and still with the attributes, but without any expectation for namedtuple functionality such as _asdict. v4.7.1 #344: Fixed regression in packages_distributions when neither top-level.txt nor a files manifest is present. v4.7.0 #330: In packages_distributions, now infer top-level names from .files() when a top-level.txt (Setuptools-specific metadata) is not present. References: python/importlib_metadata#348 python/importlib_metadata#337 python/importlib_metadata#344 python/importlib_metadata#330 (From OE-Core rev: 21d72ace8f9486bd1b478e28d53da64087d790fa) Signed-off-by: Tim Orling <[email protected]> Signed-off-by: Alexandre Belloni <[email protected]> Signed-off-by: Richard Purdie <[email protected]>

jacwalte · 2022-10-03T18:56:07Z

Looks like this issue has been reintroduced in v5.0.0.

We are seeing all pipelines fail this morning with the same message when using bandit for security linting.

Traceback (most recent call last):
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/stevedore/_cache.py", line 159, in _get_data_for_path
    with open(filename, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/python-entrypoints/3a36feec8a00a8eb17dbba25d793c41a65191f1d2d9320a96f6605c8d32530be'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/envs/.../bin/bandit", line 5, in <module>
    from bandit.cli.main import main
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/bandit/__init__.py", line 7, in <module>
    from bandit.core import config  # noqa
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/bandit/core/__init__.py", line 5, in <module>
    from bandit.core import config  # noqa
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/bandit/core/config.py", line 15, in <module>
    from bandit.core import extension_loader
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/bandit/core/extension_loader.py", line 109, in <module>
    MANAGER = Manager()
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/bandit/core/extension_loader.py", line 21, in __init__
    self.load_formatters(formatters_namespace)
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/bandit/core/extension_loader.py", line 29, in load_formatters
    verify_requirements=False,
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/stevedore/extension.py", line 136, in __init__
    verify_requirements)
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/stevedore/extension.py", line 218, in _load_plugins
    for ep in self.list_entry_points():
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/stevedore/extension.py", line 207, in list_entry_points
    eps = list(_cache.get_group_all(self.namespace))
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/stevedore/_cache.py", line 179, in get_group_all
    data = self._get_data_for_path(path)
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/stevedore/_cache.py", line 162, in _get_data_for_path
    data = _build_cacheable_data(path)
  File "/usr/local/miniconda/envs/.../lib/python3.7/site-packages/stevedore/_cache.py", line 110, in _build_cacheable_data
    for name, group_data in real_groups.items():
AttributeError: 'EntryPoints' object has no attribute 'items'

jaraco · 2022-10-03T22:56:47Z

I'm not confident of the report above for a couple of reasons.

First, the report doesn't include the actual error message. It includes a screenshot with only a FileNotFoundError.

Second, the issue reported above was addressed in 4.8.1, but the changes made in the 5.0 release were made against 5fb7029, released in 4.4.0.

Most importantly, importlib_metadata 5 still retains the tuple item access support (deprecated) added in v4.8.1.

Probably the user is reporting the issue reported in #409.

jacwalte · 2022-10-03T23:46:00Z

@jaraco I updated my comment with the full content of the error provided by the ADO (Azure Dev Ops) task.

commands that ran the task.

source $CONDA/bin/activate
python -m pip install safety==2.1.1 bandit==1.7.4
safety check
bandit --recursive .

We were not specifying the version of importlib-metadata or that it be installed here. I updated the python -m pip command to include importlib-metadata==4.13.0 and now everything is passing as expected.

new command

source $CONDA/bin/activate
python -m pip install importlib-metadata==4.13.0 safety==2.1.1 bandit==1.7.4
safety check
bandit --recursive .

Please let me know if I can provide more details.

Thanks!
Jack Walters

jaraco · 2022-10-03T23:56:44Z

Thanks. Yes, I can confirm with that traceback that it's a different issue, the one reported in #409, and that this issue could still potentially affect stevedore too, if the deprecation isn't addressed.

jaraco · 2022-10-04T15:51:46Z

@devturner Your issue is in #409.

jaraco · 2022-10-04T15:54:43Z

In 1343876, I've added a note to block the removal of the DeprecatedTuple, which deprecates the item access of an EntryPoint, until 2023-05-01 based on feedback in #409.

v4.8.1 #348: Restored support for EntryPoint access by item, deprecating support in the process. Users are advised to use direct member access instead of item-based access: - ep[0] -> ep.name - ep[1] -> ep.value - ep[2] -> ep.group - ep[:] -> ep.name, ep.value, ep.group v4.8.0 #337: Rewrote EntryPoint as a simple class, still immutable and still with the attributes, but without any expectation for namedtuple functionality such as _asdict. v4.7.1 #344: Fixed regression in packages_distributions when neither top-level.txt nor a files manifest is present. v4.7.0 #330: In packages_distributions, now infer top-level names from .files() when a top-level.txt (Setuptools-specific metadata) is not present. References: python/importlib_metadata#348 python/importlib_metadata#337 python/importlib_metadata#344 python/importlib_metadata#330 (From OE-Core rev: 21d72ace8f9486bd1b478e28d53da64087d790fa) Signed-off-by: Tim Orling <[email protected]> Signed-off-by: Alexandre Belloni <[email protected]> Signed-off-by: Richard Purdie <[email protected]>

Setuptools >71 is installed in the zuul env as a setup before running the test jobs. Recent versions of setuptools have packaging >22 as a requirement. This constraint was updated in [1]. Stevedore <= 4.8.0 is incompatible with these updates [2]. In this commit, changed stevedore upper version constraint in zuul tests to 5.0.0, matching [3]. Test Plan: pass - Mock review to update repo [4], where the constraint was causing stx-software-tox-py39 zuul job to fail pass - On the config repo, ran sysinv-tox-py39 zuul job locally, since stevedore is cited in sysinv/sysinv/sysinv/sysinv/common/utils.py Refs: [1] Updated 'packaging' python module version constraint https://review.opendev.org/c/starlingx/root/+/929167 [2] Forum post citing error message from the bug report python/importlib_metadata#348 [3] STX Openstack stevedore constraint is set to 5.0.0 https://opendev.org/starlingx/root/commit/0f050b28f2ae88e72154ba02efddb051f331aa61 [4] Mock review to the update repo to test change https://review.opendev.org/c/starlingx/update/+/929287 Closes-Bug: 2080674 Change-Id: I47ec888e8e56bcbbe547d323add9351bc540a89c Signed-off-by: Leonardo Fagundes Luz Serrano <[email protected]>

mr-c changed the title ~~#339 / v4.8.0 broke stevedore which broke bandit~~ #339 / v4.8.0 broke stevedore which broke bandit & openstackclient Aug 29, 2021

mtreinish mentioned this issue Aug 29, 2021

Pin importlib-metadata in CI Qiskit/qiskit#6961

Closed

mtreinish mentioned this issue Aug 29, 2021

Pin importlib-metadata in CI Qiskit/rustworkx#427

Closed

mr-c changed the title ~~#339 / v4.8.0 broke stevedore which broke bandit & openstackclient~~ #339 / v4.8.0 broke stevedore which broke bandit, openstackclient, flake8 and others Aug 29, 2021

jaraco added a commit that referenced this issue Aug 29, 2021

Restore support for EntryPoint access by item. Fixes #348.

205a9bc

jaraco added a commit that referenced this issue Aug 29, 2021

Restore support for EntryPoint access by item. Fixes #348.

a6a2f58

jaraco added a commit that referenced this issue Aug 29, 2021

Restore support for EntryPoint access by item. Fixes #348.

177d550

jaraco mentioned this issue Aug 29, 2021

Restore support for EntryPoint access by item. #349

Merged

jaraco mentioned this issue Aug 29, 2021

Rely on member access, the preferred access since importlib_metadata 4.8. openstack/stevedore#4

Closed

jaraco closed this as completed in #349 Aug 29, 2021

jaraco mentioned this issue Aug 29, 2021

Complete type annotations #342

Closed

This comment was marked as off-topic.

Sign in to view

jaraco added a commit that referenced this issue Oct 4, 2022

Add minimum retention of DeprecatedTuple. Ref #409, Ref #348.

1343876

ShyftXero mentioned this issue Mar 28, 2023

pip install didn't work. hellman/xortool#44

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#339 / v4.8.0 broke stevedore which broke bandit, openstackclient, flake8 and others #348

#339 / v4.8.0 broke stevedore which broke bandit, openstackclient, flake8 and others #348

mr-c commented Aug 29, 2021

Stannislav commented Aug 29, 2021 •

edited

Loading

freedge commented Aug 29, 2021

WilliamDEdwards commented Aug 29, 2021 •

edited

Loading

jaraco commented Aug 29, 2021 •

edited

Loading

jaraco commented Aug 29, 2021

jaraco commented Aug 29, 2021

mr-c commented Aug 29, 2021

jaraco commented Aug 29, 2021

jaraco commented Aug 29, 2021

jacwalte commented Oct 3, 2022 •

edited by jaraco

Loading

jaraco commented Oct 3, 2022 •

edited

Loading

jacwalte commented Oct 3, 2022

jaraco commented Oct 3, 2022

This comment was marked as off-topic.

jaraco commented Oct 4, 2022

jaraco commented Oct 4, 2022

#339 / v4.8.0 broke stevedore which broke bandit, openstackclient, flake8 and others #348

#339 / v4.8.0 broke stevedore which broke bandit, openstackclient, flake8 and others #348

Comments

mr-c commented Aug 29, 2021

Stannislav commented Aug 29, 2021 • edited Loading

freedge commented Aug 29, 2021

WilliamDEdwards commented Aug 29, 2021 • edited Loading

jaraco commented Aug 29, 2021 • edited Loading

jaraco commented Aug 29, 2021

jaraco commented Aug 29, 2021

mr-c commented Aug 29, 2021

jaraco commented Aug 29, 2021

jaraco commented Aug 29, 2021

jacwalte commented Oct 3, 2022 • edited by jaraco Loading

jaraco commented Oct 3, 2022 • edited Loading

jacwalte commented Oct 3, 2022

jaraco commented Oct 3, 2022

This comment was marked as off-topic.

jaraco commented Oct 4, 2022

jaraco commented Oct 4, 2022

Stannislav commented Aug 29, 2021 •

edited

Loading

WilliamDEdwards commented Aug 29, 2021 •

edited

Loading

jaraco commented Aug 29, 2021 •

edited

Loading

jacwalte commented Oct 3, 2022 •

edited by jaraco

Loading

jaraco commented Oct 3, 2022 •

edited

Loading