[BUG] Using canonicalize_version from _vendor instead of extern breaks packages

[armurox]

setuptools version

setuptools==71.0.0, setuptools==71.0.1, setuptools==71.0.3

Python version

Python 3.10.0

OS

Ubuntu

Additional environment information

This is happening when installing internal packages at my firm which I cannot share, but am happy to get answer general questions.

Description

Since version 71.0.0 release, some of our services which depend on internal packages have been breaking, and all the errors are precisely this one:

Appears to be an issue with moving from .extern.packaging.utils.canonicalize_versions to setuptools .vendor.packaging.utils.canonicalize_version (as that is the change from version 70.3.0 -> 71.0.0 from what I can see in the setuptools diff, related to canonicalize_version).

Expected behavior

This happened while installing the internal packages in the docker environment for our service.

How to Reproduce

Steps to reproduce are a little difficult here, as I cannot provide the internal package.

Output

File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/__init__.py", line 106, in setup
#12 15.35           return distutils.core.setup(**attrs)
#12 15.35         File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 184, in setup
#12 15.35           return run_commands(dist)
#12 15.35         File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
#12 15.35           dist.run_commands()
#12 15.35         File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 970, in run_commands
#12 15.35           self.run_command(cmd)
#12 15.35         File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/dist.py", line 974, in run_command
#12 15.35           super().run_command(command)
#12 15.35         File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 989, in run_command
#12 15.35           cmd_obj.run()
#12 15.35         File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 321, in run
#12 15.35           self.find_sources()
#12 15.35         File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
#12 15.35           mm.run()
#12 15.35         File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 555, in run
#12 15.35           self.prune_file_list()
#12 15.35         File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 621, in prune_file_list
#12 15.35           base_dir = self.distribution.get_fullname()
#12 15.35         File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/_core_metadata.py", line 266, in get_fullname
#12 15.35           return _distribution_fullname(self.get_name(), self.get_version())
#12 15.35         File "/home/dubizzle/venv/lib/python3.10/site-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
#12 15.35           canonicalize_version(version, strip_trailing_zero=False),
#12 15.35       TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
#12 15.35       [end of output]
#12 15.35   

[cbm755]

I have a public project that exhibits this. My downstream issue: https://gitlab.com/plom/plom/-/issues/3491

CI failure example: https://gitlab.com/plom/plom/-/jobs/7385812246

(Haven't spent any time looking at what's happening here other than it started when 71.0.0 came out and still happens on 71.0.3).

[ben-xD]

Simple example doesn't reproduce bug

I couldn't reproduce it when just doing it on a simple setup.

• docker run -it ubuntu:22.04 bash
• Inside the shell:
  • install python: apt install -y curl python3.11
  • apt-get install python3-pip
  • install setuptools: pip3 install --user -U pip setuptools wheel
  • confirm setuptools is the new one in a python repl: import setuptools, then setuptools.__version__ prints 71.0.3
• try to install empy: pip install empy, and it works. I made sure to use the same version of python (3.11.8), pip, wheel, setuptools as in the stacktrace of the failing version.

I also have a more complex dockerfile I can't share, that has the issue. Here's the stacktrace anyway:

reproducible example (although more complicated code, involving another projects scripts)

• clone the repo: git clone [email protected]:ben-xD/ardupilot-sitl-docker.git
• init submodule: git submodule update --recursive
• start build: docker-compose build --no-cache local

stack trace of bug

202.3 + pip3 install --user -U pip setuptools wheel
202.6 Requirement already satisfied: pip in /usr/lib/python3/dist-packages (22.0.2)
202.7 Collecting pip
202.8   Downloading pip-24.1.2-py3-none-any.whl (1.8 MB)
203.1      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 6.3 MB/s eta 0:00:00
203.1 Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (59.6.0)
203.3 Collecting setuptools
203.3   Downloading setuptools-71.0.3-py3-none-any.whl (2.3 MB)
203.6      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 7.2 MB/s eta 0:00:00
203.7 Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (0.37.1)
203.7 Collecting wheel
203.7   Downloading wheel-0.43.0-py3-none-any.whl (65 kB)
203.7      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.8/65.8 KB 8.2 MB/s eta 0:00:00
203.8 Installing collected packages: wheel, setuptools, pip
203.8   WARNING: The script wheel is installed in '/home/docker/.local/bin' which is not on PATH.
203.8   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
204.9   WARNING: The scripts pip, pip3 and pip3.10 are installed in '/home/docker/.local/bin' which is not on PATH.
204.9   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
204.9 Successfully installed pip-24.1.2 setuptools-71.0.3 wheel-0.43.0
204.9 + '[' '' == true ']'
204.9 + '[' jammy == bookworm ']'
204.9 + '[' jammy == lunar ']'
204.9 + '[' jammy == mantic ']'
204.9 + '[' jammy == noble ']'
204.9 + pip3 install --user -U future lxml pymavlink pyserial MAVProxy pexpect geocoder empy==3.3.4 ptyprocess dronecan flake8 junitparser pygame intelhex
205.2 Collecting future
205.2   Downloading future-1.0.0-py3-none-any.whl.metadata (4.0 kB)
205.2 Requirement already satisfied: lxml in /usr/lib/python3/dist-packages (4.8.0)
205.4 Collecting lxml
205.4   Downloading lxml-5.2.2-cp310-cp310-manylinux_2_28_aarch64.whl.metadata (3.4 kB)
205.5 Collecting pymavlink
205.5   Downloading pymavlink-2.4.41-py3-none-any.whl.metadata (6.2 kB)
205.5 Requirement already satisfied: pyserial in /usr/lib/python3/dist-packages (3.5)
205.6 Collecting MAVProxy
205.6   Downloading MAVProxy-1.8.70-py3-none-any.whl.metadata (1.5 kB)
205.7 Collecting pexpect
205.7   Downloading pexpect-4.9.0-py2.py3-none-any.whl.metadata (2.5 kB)
205.8 Collecting geocoder
205.8   Downloading geocoder-1.38.1-py2.py3-none-any.whl.metadata (14 kB)
205.9 Collecting empy==3.3.4
205.9   Downloading empy-3.3.4.tar.gz (62 kB)
206.0      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.9/62.9 kB 3.7 MB/s eta 0:00:00
206.0   Preparing metadata (setup.py): started
206.1   Preparing metadata (setup.py): finished with status 'error'
206.1   error: subprocess-exited-with-error
206.1
206.1   × python setup.py egg_info did not run successfully.
206.1   │ exit code: 1
206.1   ╰─> [28 lines of output]
206.1       Traceback (most recent call last):
206.1         File "<string>", line 2, in <module>
206.1         File "<pip-setuptools-caller>", line 34, in <module>
206.1         File "/tmp/pip-install-xc_143b0/empy_9aaf07e0bf9b46e5a7db29818d00f106/setup.py", line 24, in <module>
206.1           setup(
206.1         File "/home/docker/.local/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 184, in setup
206.1           return run_commands(dist)
206.1         File "/home/docker/.local/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
206.1           dist.run_commands()
206.1         File "/home/docker/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 970, in run_commands
206.1           self.run_command(cmd)
206.1         File "/home/docker/.local/lib/python3.10/site-packages/setuptools/dist.py", line 974, in run_command
206.1           super().run_command(command)
206.1         File "/home/docker/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 989, in run_command
206.1           cmd_obj.run()
206.1         File "/home/docker/.local/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 321, in run
206.1           self.find_sources()
206.1         File "/home/docker/.local/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
206.1           mm.run()
206.1         File "/home/docker/.local/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 555, in run
206.1           self.prune_file_list()
206.1         File "/home/docker/.local/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 621, in prune_file_list
206.1           base_dir = self.distribution.get_fullname()
206.1         File "/home/docker/.local/lib/python3.10/site-packages/setuptools/_core_metadata.py", line 266, in get_fullname
206.1           return _distribution_fullname(self.get_name(), self.get_version())
206.1         File "/home/docker/.local/lib/python3.10/site-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
206.1           canonicalize_version(version, strip_trailing_zero=False),
206.1       TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
206.1       [end of output]
206.1
206.1   note: This error originates from a subprocess, and is likely not a problem with pip.
206.1 error: metadata-generation-failed
206.1
206.1 × Encountered error while generating package metadata.
206.1 ╰─> See above for output.
206.1
206.1 note: This is an issue with the package mentioned above, not pip.
206.1 hint: See above for details.

[cbm755]

[while typing this @ben-xD posted another example]

I started trying to make a MWE. But instead here's an example that I think is roughly equivalent to my project but it doesn't fail... more digging required:

podman run -it --rm ubuntu:22.04 bash   # or docker
apt update
apt install python3-dev python3-pip
pip install -U setuptools
pip install -U canvasapi 

[cbm755]

Ok I can reproduce it without my full project like this:

podman pull ubuntu:22.04

podman run -it --rm ubuntu:22.04 bash

apt update
DEBIAN_FRONTEND=noninteractive apt-get install -y tzdata curl

apt-get --no-install-recommends --yes install \
      cmake make g++ imagemagick \
      dvipng latexmk texlive-latex-extra texlive-fonts-recommended \
      libgl1-mesa-glx libsm6 libxrender1 libegl1 libxkbcommon0 \
      libdbus-1-3 libpango-1.0-0 libpangocairo-1.0-0 \
      python3-pytest python3-dev python3-pip \
      iproute2 psmisc file python3-magic

pip install -U setuptools wheel pip

pip install canvasapi 

which dies like this:

Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (59.6.0)
Collecting setuptools
  Downloading setuptools-71.0.3-py3-none-any.whl (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 8.3 MB/s eta 0:00:00
Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (0.37.1)
Collecting wheel
  Downloading wheel-0.43.0-py3-none-any.whl (65 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.8/65.8 KB 9.8 MB/s eta 0:00:00
Requirement already satisfied: pip in /usr/lib/python3/dist-packages (22.0.2)
Collecting pip
  Downloading pip-24.1.2-py3-none-any.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 9.5 MB/s eta 0:00:00
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: wheel
    Found existing installation: wheel 0.37.1
    Not uninstalling wheel at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'wheel'. No files were found to uninstall.
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.6.0
    Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'setuptools'. No files were found to uninstall.
  Attempting uninstall: pip
    Found existing installation: pip 22.0.2
    Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-24.1.2 setuptools-71.0.3 wheel-0.43.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Collecting canvasapi
  Downloading canvasapi-3.2.0.tar.gz (87 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.0/87.0 kB 2.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [40 lines of output]
      running egg_info
      creating /tmp/pip-pip-egg-info-ak9nqcsg/canvasapi.egg-info
      writing /tmp/pip-pip-egg-info-ak9nqcsg/canvasapi.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-ak9nqcsg/canvasapi.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-pip-egg-info-ak9nqcsg/canvasapi.egg-info/requires.txt
      writing top-level names to /tmp/pip-pip-egg-info-ak9nqcsg/canvasapi.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-ak9nqcsg/canvasapi.egg-info/SOURCES.txt'
      reading manifest file '/tmp/pip-pip-egg-info-ak9nqcsg/canvasapi.egg-info/SOURCES.txt'
      adding license file 'LICENSE'
      adding license file 'AUTHORS.md'
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-lgjbyevy/canvasapi_441acf653eaa4aafbcaaf84078bf172f/setup.py", line 20, in <module>
          setup(
        File "/usr/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
        File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
        File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 970, in run_commands
          self.run_command(cmd)
        File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 974, in run_command
          super().run_command(command)
        File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/usr/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/usr/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/usr/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 555, in run
          self.prune_file_list()
        File "/usr/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 621, in prune_file_list
          base_dir = self.distribution.get_fullname()
        File "/usr/local/lib/python3.10/dist-packages/setuptools/_core_metadata.py", line 266, in get_fullname
          return _distribution_fullname(self.get_name(), self.get_version())
        File "/usr/local/lib/python3.10/dist-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
          canonicalize_version(version, strip_trailing_zero=False),
      TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

I'll see if I can narrow down that package list a bit more...

[cbm755]

Narrowed it down to this MWE:

podman run -it --rm ubuntu:22.04 bash   # or docker
apt update
apt-get -y install python3-dev python3-pip 

apt-get -y install  python3-pytest    # works w/o, crashes with

pip install -U setuptools wheel pip
pip install canvasapi 

[mikealfare]

@cbm755 We ran into this too. The issue for us was that packaging<24.0 was installed. setuptools>=71.0 does not work with packaging<24.0, but will not upgrade it if packaging is already installed. pytest takes a runtime dependency on packaging. What version of packaging do you have after installing python3-pytest?

Disclosure: I'm just a random person on the internet and have nothing to do with pypa or setuptools. I'm just trying to help.

[cbm755]

Thanks @mikealfare yes my MWE has packaging version 21.1. pip install -U packaging makes it work. Thanks, that'll probably get my CI running again as was having trouble getting setuptools pinned down before 71 everywhere.

[jaraco]

This issue is a duplicate of #4483. I'll pin that issue for extra visibility.

[jlevitt]

@cbm755 We ran into this too. The issue for us was that packaging<24.0 was installed. setuptools>=71.0 does not work with packaging<24.0, but will not upgrade it if packaging is already installed. pytest takes a runtime dependency on packaging. What version of packaging do you have after installing python3-pytest?

Disclosure: I'm just a random person on the internet and have nothing to do with pypa or setuptools. I'm just trying to help.

This fixed the issue for me, thanks!