pip's vendored pkg_resources should stop using pkgutil.ImpImporter

[warsaw]

Description

pkgutil.ImpImporter has been deprecated since Python 3.3. It is slated to be removed in Python 3.12 but until pip no longer uses it, we probably can't remove this long deprecated class.

Expected behavior

No response

pip version

22.2.2

Python version

3.12

OS

macOS

How to Reproduce

1. Grab the CPython branch: gh pr checkout 98059
2. configure && make test
3. test_venv fails because pip uses this API
4. See https://github.com/python/cpython/actions/runs/3215689824/jobs/5256965254

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[warsaw]

pip also needs to remove its reference to importer.find_module() as that API is also slated for removal.

[pradyunsg]

Thanks for filing this @warsaw!

This module is under _vendor, so... it comes from a different project that pip uses. Looking into this, they're still referenced in upstream setuptools:

• https://github.com/pypa/setuptools/blob/2d4ebde71ce9daf914318f5ff29bcc39507b2cdf/pkg_resources/__init__.py#L2191
• https://github.com/pypa/setuptools/blob/2d4ebde71ce9daf914318f5ff29bcc39507b2cdf/pkg_resources/__init__.py#L2232

I believe this needs to get fixed there, and we'll pick up the fix once it's made there. :)

[pradyunsg]

@warsaw Could you file an issue against https://github.com/pypa/setuptools for this?

[pradyunsg]

/cc @jaraco for visibility, since setuptools "owns" pkg_resources and would need to make the relevant changes on this front.

[pradyunsg]

/cc @pfmoore for awareness as the current RM -- to be clear, I don't know this affects any of the plans for the current release. I'm flagging this for awareness since there is, maybe, a chance that there's a setuptools release after we make this time's round of updates and before the release, in which case we might want to run another vendoring update. :)

[warsaw]

@warsaw Could you file an issue against https://github.com/pypa/setuptools for this?

Thanks @pradyunsg; I'll do this, although I'm not sure how to figure out which version of setuptools pip is currently vendoring. I'll just file this against the latest, i.e. 65.4.1

[warsaw]

See the referenced setuptools issue above. And thanks for working on this!

[pradyunsg]

I'm not sure how to figure out which version of setuptools pip is currently vendoring.

So... this issue made me realise that I'd broken our update logic in a subtle manner, which means we're on a ~2 year old version.

pip/src/pip/_vendor/vendor.txt

Line 19 in 8f381cd

setuptools==44.0.0

#11500 (comment) is where I noticed that. I'll be trying to bump to the latest setuptools version in the next pip release, as a part of #11502 (or a follow up); assuming it's tractable in the limited free time that I have. :)

[uranusjr]

Since we are already moving from pkg_resources to importlib.metadata, we can make Python 3.12 our cutoff to switch entirely to the latter backend. This would mean that we can simply keep pkg_resources as-is since those usages will only be accessed on earlier Python versions.

[uranusjr]

Does this need further work after #11689 is successfully implemented?

[hroncok]

The change landed in setuptools 66.1, so I'd say not yet.

[warsaw]

What are the chances of updating the pkg_resources vendor and closing this, with a new release of pip? I thought we'd plumbed this all the way through the toolchain so that the vendored pip in CPython main would only conditionally use deprecated APIs, but that doesn't seem to be the case. CPython still fails with the vendored pip 23.1.1 in my remove-imp branch.

[uranusjr]

We tried that (#11689), it introduced other issues and got reverted (#11976).

[warsaw]

So, I guess we're at an impasse. We can't remove these long deprecated APIs without fixing the stack of dependencies vendored into CPython. Time's also getting short for Python 3.12 and it will be sad if we can't make progress on this for another year. Any thoughts on how to proceed?

[pfmoore]

I'll work on this.

[pfmoore]

#11997

[pfmoore]

OK, the fix is now merged and I'll do the release tomorrow (UK time) - in 12 hours or so. Hopefully, that will be the last iteration we'll need on all of this! Thanks everyone for the discussion and your patience!

[pfmoore]

Pip 23.1.2 has now been released.

[bersbersbers]

Reading the above discussion, I am unsure if this should be considered fixed. Either way, I am using pip 23.1.2 on Python 3.12.0b1 via pyenv on Windows 10, and I am unable to install matplotlib or numpy:

pip install numpy
Collecting numpy
  Using cached numpy-1.24.3.tar.gz (10.9 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      Traceback (most recent call last):
        File "C:\Users\bers\.pyenv-win-venv\envs\project_3.12\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "C:\Users\bers\.pyenv-win-venv\envs\project_3.12\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\bers\.pyenv-win-venv\envs\project_3.12\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 112, in get_requires_for_build_wheel
          backend = _build_backend()
                    ^^^^^^^^^^^^^^^^
        File "C:\Users\bers\.pyenv-win-venv\envs\project_3.12\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 77, in _build_backend
          obj = import_module(mod_path)
                ^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\bers\.pyenv\pyenv-win\versions\3.12.0b1\Lib\importlib\__init__.py", line 90, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen importlib._bootstrap>", line 1293, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1266, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1216, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 400, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1293, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1266, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1237, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 841, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 994, in exec_module
        File "<frozen importlib._bootstrap>", line 400, in _call_with_frames_removed
        File "C:\Users\bers\AppData\Local\Temp\pip-build-env-8dt1uf4c\overlay\Lib\site-packages\setuptools\__init__.py", line 16, in <module>
          import setuptools.version
        File "C:\Users\bers\AppData\Local\Temp\pip-build-env-8dt1uf4c\overlay\Lib\site-packages\setuptools\version.py", line 1, in <module>
          import pkg_resources
        File "C:\Users\bers\AppData\Local\Temp\pip-build-env-8dt1uf4c\overlay\Lib\site-packages\pkg_resources\__init__.py", line 2172, in <module>
          register_finder(pkgutil.ImpImporter, find_on_path)
                          ^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I have done the tour via pypa/setuptools#3631 and #11688 all the way to here, which is the only issue still open.

[uranusjr]

This looks like a setuptools issue. pip does not import setuptools unless the package tells it to, so this is something you should report to either setuptools or the packages using it and triggering the error.

[abravalheri]

I suspect that setuptools has already fixed problems with pkgutil.ImpImporter in v66.1.0.

To make sure the package builds on Python 3.12, the best would be to use a relatively recent version of setuptools. This is probably specified in the pyproject.toml file of the projects you are trying to build.

[bersbersbers]

This is probably specified in the pyproject.toml file of the projects you are trying to build.

Ah, this seems to be the key thing, because I have setuptools 67.8.0 installed but that is irrelevant.

So this explains why numpy does not install:
https://github.com/numpy/numpy/blob/cc0abd768575d7f9e862de0b4912af27f6e9690d/pyproject.toml#LL7C6-L7C16

But this does not explain why matplotlib does not install, does it?
https://github.com/matplotlib/matplotlib/blob/7b821ce6b4a4474b1013435954c05c63574e3814/pyproject.toml#L7

[abravalheri]

I am not sure how the dependency resolution algorithm works during build time, but matplotlib seems to depend on numpy, there might be a chance the same environment is being used to build both and then the same version of setuptools would be used?

@uranusjr might be more familiar with the dependency resolution at this stage...

[bersbersbers]

matplotlib seems to depend on numpy

You are right - even at build time:
https://github.com/matplotlib/matplotlib/blob/7b821ce6b4a4474b1013435954c05c63574e3814/pyproject.toml#LL5C29-L5C29

Okay, that explains that as well. I guess I will open an issue with numpy then.

[pradyunsg]

With #11997 landed, is there anything more needed here other than a pip release (and it's in 23.1.2)?

[ghost]

How do I fix pip not being able to install anything or do pip freeze. I have faced it several times, where in python3.12 pip is working normally but after some commands, it doesnt work and cpython have to reinstalled through git clone. python/cpython#106824

[uranusjr]

We already fixed this in pip 23.2 actually (since the bundled pkg_resources has been upgraded to 67.7.1). The ensurepip upgrade will happen soon.

(Edit: the bundled pip in CPython has since been updated to 23.2.1 on 2023-07-25. See python/cpython@f443b54)