Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

5.0.0 is causing problems in Python 3.7. Can't import celery until I downgrade to 4.13.0. #411

Closed
coredumperror opened this issue Oct 5, 2022 · 11 comments

Comments

@coredumperror
Copy link

I can't run the import statement from celery import Celery in Python 3.7 with importlib-metadata 5.0.0 installed. I get:

>>> from celery import Celery
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Celery' from 'celery' (/Users/rrollins/.pyenv/versions/temp-3.7/lib/python3.7/site-packages/celery/__init__.py)

This makes no sense. That file totally has a Celery object in it... kindof. It's brought in by an unusual construction at the bottom of https://github.com/celery/celery/blob/master/celery/__init__.py, which may be what's confusing importlib-metadata 5.

Minimally necessary install to trigger this is to make a virtualenv from Python 3.7.10, activate it, and run pip install celery. This will automatically install importlib-metadata 5.0.0.

Then just start python and run from celery import Celery. It'll crash with the above traceback. Downgrade to importlib-metadata==4.13.0 and it won't crash.

This also affects 3.7.13, so it probably affects all other point releases of 3.7. We use 3.7.10 because that's what the AmazonLinux docker image provides.

Oddly enough, installing celery on Python 3.8 doesn't result in importlib-metadata being installed automatically. The import works fine without it installed. And manually installing 5.0 doesn't break the import like it does in 3.7. So this seems to be a Python 3.7-specific issue.

@TkTech
Copy link

TkTech commented Oct 6, 2022

Can confirm this was a very confusing bug that hit us today, and we tracked it down to an update of importlib-metadata from 4.12.0 to 5.0.0. From a quick check it seems to be this line, v4.13.0...v5.0.0#diff-bf79a43449f7a7e1e76063e303fbdd35bec7eb50f2e9ddba26e3048def32ed06R856, but it seems odd this would impact importing celery.

TkTech added a commit to TkTech/celery-heimdall that referenced this issue Oct 6, 2022
@sonerayberk
Copy link

Same issue.

@Safihre
Copy link

Safihre commented Oct 7, 2022

Yep same here! Breaks Python 3.7 compatibility.
Took a long time to figure out....

@longwuyuan
Copy link

This broke every single airflow cluster that did a pull :-(

@Safihre
Copy link

Safihre commented Oct 15, 2022

See pinned issue #409.
@jaraco can you confirm if this really is Python 3.7 specific problem or the problem is in the packages we use that need updating?

@jaraco
Copy link
Member

jaraco commented Oct 15, 2022

See celery/celery#7783 where celery has been tracking the issue.

confirm if this really is Python 3.7 specific problem or the problem is in the packages we use that need updating?

The issue isn't specific to Python 3.7. It will affect any consumers of importlib_metadata 5 or later that use entry_points() and rely on the dict-like interface it previously returned. In the case of Celery, it did only affect Python 3.7 and earlier because it relied on stdlib's importlib.metadata on Python 3.8 and later.

There are many projects that only rely on importlib_metadata for Python 3.7 and earlier and rely on importlib.metadata in the stdlib for 3.8 and later. These projects are also at risk, because in Python 3.12, importlib.metadata gets the same behavior. Only on importlib_metadata 3.6 and Python 3.10 or later is the SelectableGroups compatibility interface supplied, so anyone using importlib.metadata on Python 3.8 or 3.9 must use the deprecated interface.

That's why I recommend to either:

Other options that I don't recommend include:

  • update the code to adapt different interfaces on Python 3.8 and 3.9
  • Bypass the entry_points function entirely and use more raw interfaces on Distribution objects.

A temporary workaround is to pin to importlib_metadata < 5 (or force-install that version).

I recognize that as consumers of Celery, this issue feels out of your control, which is why I'm glad to see that Celery has addressed the issue (though best I can tell, the fix isn't released yet).

Reliance on this deprecated behavior has been marked as Deprecated for over a year. I'm curious to see - is there something unique to celery (or other projects) that made it difficult for these downstream projects to detect and respond to the warnings?

I can confirm that in at least one run, celery did emit the warnings:

    =============================== warnings summary ===============================
  .tox/3.7-integration-redis/lib/python3.7/site-packages/kombu/utils/compat.py:85
    /home/runner/work/celery/celery/.tox/3.7-integration-redis/lib/python3.7/site-packages/kombu/utils/compat.py:85: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
      entry_points = importlib_metadata.entry_points().get(namespace, [])
  
  t/integration/test_canvas.py: 150 warnings
  t/integration/test_inspect.py: 5 warnings
  t/integration/test_security.py: 1 warning
  t/integration/test_tasks.py: 40 warnings
    /home/runner/work/celery/celery/celery/utils/imports.py:144: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
      for ep in entry_points().get(namespace, []):

Things that celery and other projects can do to detect these incoming incompatible changes:

  • When testing, treat warnings like errors. This approach means more maintenance burden as any new warning will result in a broken CI. On the other hand, it creates a strong signal that action is required. The errors can be triaged, registered in a bug tracker, and suppressed, and the maintainers can prioritize the fix as needed.
  • Detect warnings when testing interactively. Tools like pytest will change the result from green to yellow if any warnings are issued. Anything but a green result indicates that action is required.
  • Create a separate workflow or alerting system for detecting and reporting warnings.

I regret the disruption caused here. If there's something more that importlib_metadata can do, please let me know.

@jaraco jaraco closed this as completed Oct 15, 2022
@Safihre
Copy link

Safihre commented Oct 17, 2022

@jaraco Thanks for the details.
Just wanted to let you know that in our runs, we haven't seen anything.
For example a run where I have set importlib_metadata<5.0.0.
The problem for us was in tavern which relies on stevedore.

Safihre added a commit to sabnzbd/sabnzbd that referenced this issue Oct 17, 2022
@jaraco
Copy link
Member

jaraco commented Oct 17, 2022

@jaraco Thanks for the details. Just wanted to let you know that in our runs, we haven't seen anything. For example a run where I have set importlib_metadata<5.0.0. The problem for us was in tavern which relies on stevedore.

I looked briefly and I concur - it appears as if no warning was issued for sabnzbd/sabnzbd and I don't see anything that would have disabled the warnings (e.g. -W ignore). I think it would be a useful exercise to investigate why the tests failed to report the deprecation warning, as it may help me understand a condition where such a warning doesn't reach the intended audience, or it may help projects like yours avoid this situation for other deprecations.

Is it possible that stevedore has updated the underlying usage and that's why the warnings aren't emitted any longer? Are you able to run with -W error to force any deprecation warnings to raise as Exceptions to find a traceback showing the codepath that's reached?

@jaraco
Copy link
Member

jaraco commented Oct 17, 2022

I suspect the way tavern.run invokes pytest.main directly may be implicated in masking any warnings that occur within that run.

Balandat added a commit to Balandat/tutorials that referenced this issue Oct 20, 2022
v5.0.0 causes issues with python 3.7: python/importlib_metadata#411

This was raised in the context of pytorch#2090

This PR pins the version to <5.0 to circumvent this. Maybe a better fix would be to bump the python version in CI to 3.8, but that's something to discuss more broadly.
Balandat added a commit to pytorch/tutorials that referenced this issue Oct 21, 2022
svekars pushed a commit to pytorch/tutorials that referenced this issue Oct 21, 2022
vkuzo added a commit to pytorch/tutorials that referenced this issue Oct 22, 2022
* update NS for FX tutorial for PyTorch v1.13

Summary:

Makes a couple of updates to ensure this tutorial still runs on 1.13:
1. changes the `qconfig_dict` argument of `prepare_fx` to `qconfig_mapping`
2. adds `example_inputs` to `prepare_fx`

Test plan:

Run the tutorial, it runs without errors on master

* Pin importlib_metadata<5.0 for python <= 3.7 in requirements.txt (#2091)

v5.0.0 causes issues with python 3.7: python/importlib_metadata#411

* Enable the FX tutorial

Co-authored-by: Max Balandat <[email protected]>
Co-authored-by: Svetlana Karslioglu <[email protected]>
fredkingham pushed a commit to openhealthcare/opal that referenced this issue Nov 29, 2022
fredkingham pushed a commit to openhealthcare/opal that referenced this issue Nov 29, 2022
@Lawouach
Copy link

Lawouach commented Apr 3, 2023

Hi @jaraco

I failed to detect this issue with the Chaos Toolkit but an user reported it recently.

import click
import importlib_metadata
from click_plugins import with_plugins

@click.group
def cli():
    pass


with_plugins(importlib_metadata.entry_points().get("chaostoolkit.cli_plugins"))(cli)

This throws an error with importlib_metadata >= 5

So I switched to:

import click
import importlib_metadata
from click_plugins import with_plugins

@click.group
def cli():
    pass

with_plugins(importlib_metadata.entry_points(group="chaostoolkit.cli_plugins"))(cli)

This now works.

But I'm confused by your statement above. What's the right path forward? Should I use this fix and force importlib_metadata >= 6 ? Or should I stick to older versions as other projects will likely not accept this newer version when installing?

Basically, I'm not clear what's the official position.

Cheers,

Lawouach added a commit to chaostoolkit/chaostoolkit that referenced this issue Apr 3, 2023
@jaraco
Copy link
Member

jaraco commented Apr 11, 2023

You don't need to force importlib_metadata 6. >=3.6 is the minimum version for selectable groups (the latter usage). If you can't rely on that version (released Feb 21), there is backports.entry_points_selectable that provides the preferred interface for older importlib_metadata and Python releases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants