Pipenv install of Python 3.9 fails using ubuntu AMI

[david-mcclure]

Issue description

We have a solution deployed on AWS where an EC2 instance uses docker files to build images for different components. One of those components has a docker script we haven't changed in over a year that, among other things, installs python 3.9 using pipenv. We ran it as recently as two days ago without any problems. When we tried running today, we got an error we've never seen before from the pipenv install.

Any help would be greatly appreciated. Thanks!

Expected result

When script runs "RUN pipenv install --python 3.9" that it succeeds. See log from last successful run:

15:02:54 #15 [11/12] RUN pipenv install --python 3.9
15:02:55 #15 0.914 Warning: the environment variable LANG is not set!
15:02:55 #15 0.914 We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.
15:02:55 #15 1.290 Creating a virtualenv for this project...
15:02:55 #15 1.290 Pipfile: /code/Pipfile
15:02:55 #15 1.316 Using /usr/bin/python3.9 (3.9.5) to create virtualenv...
15:02:56 #15 2.101 created virtual environment CPython3.9.5.final.0-64 in 476ms
15:02:56 #15 2.101 creator CPython3Posix(dest=/root/.local/share/virtualenvs/code-_Py8Si6I, clear=False, no_vcs_ignore=False, global=False)
15:02:56 #15 2.101 seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
15:02:56 #15 2.101 added seed packages: pip==24.1, setuptools==70.1.0, wheel==0.43.0
15:02:56 #15 2.101 activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
15:02:56 #15 2.101
15:02:56 #15 2.102 ✔ Successfully created virtual environment!
15:02:56 #15 2.157 Virtualenv location: /root/.local/share/virtualenvs/code-_Py8Si6I
15:02:56 #15 2.208 To activate this project's virtualenv, run pipenv shell.
15:02:56 #15 2.208 Alternatively, run a command inside the virtualenv with pipenv run.
15:02:56 #15 2.220 To activate this project's virtualenv, run pipenv shell.
15:02:56 #15 2.220 Alternatively, run a command inside the virtualenv with pipenv run.
15:02:56 #15 2.223 Installing dependencies from Pipfile.lock (d65a77)...

Actual result

When script runs "RUN pipenv install --python 3.9", it fails. See log from last run:

15:49:43 #15 [11/12] RUN pipenv install --python 3.9
15:49:44 #15 0.935 Warning: the environment variable LANG is not set!
15:49:44 #15 0.935 We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.
15:49:45 #15 1.345 Creating a virtualenv for this project...
15:49:45 #15 1.346 Pipfile: /code/Pipfile
15:49:45 #15 1.387 Using /usr/bin/python3.9 (3.9.5) to create virtualenv...
15:49:45 #15 1.767
15:49:45 #15 1.767 ✘ Failed creating virtual environment
15:49:45 #15 1.767 [pipenv.exceptions.VirtualenvCreationException]: Traceback (most recent call last):
15:49:45 #15 1.767 File "/usr/lib/python3.9/runpy.py", line 188, in _run_module_as_main
15:49:45 #15 1.767 mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
15:49:45 #15 1.767 File "/usr/lib/python3.9/runpy.py", line 147, in _get_module_details
15:49:45 #15 1.767 return _get_module_details(pkg_main_name, error)
15:49:45 #15 1.767 File "/usr/lib/python3.9/runpy.py", line 111, in _get_module_details
15:49:45 #15 1.767 import(pkg_name)
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/init.py", line 3, in
15:49:45 #15 1.767 from .run import cli_run, session_via_cli
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/run/init.py", line 15, in
15:49:45 #15 1.767 from .plugin.creators import CreatorSelector
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/run/plugin/creators.py", line 7, in
15:49:45 #15 1.767 from virtualenv.create.via_global_ref.builtin.builtin_way import VirtualenvBuiltin
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/create/via_global_ref/builtin/builtin_way.py", line 5, in
15:49:45 #15 1.767 from virtualenv.create.creator import Creator
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/create/creator.py", line 13, in
15:49:45 #15 1.767 from virtualenv.discovery.cached_py_info import LogCmd
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/discovery/cached_py_info.py", line 25, in
15:49:45 #15 1.767 _CACHE[Path(sys.executable)] = PythonInfo()
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/discovery/py_info.py", line 100, in init
15:49:45 #15 1.767 self.distutils_install = self._distutils_install().copy()
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/virtualenv/discovery/py_info.py", line 191, in _distutils_install
15:49:45 #15 1.767 i.finalize_options()
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/command/install.py", line 57, in finalize_options
15:49:45 #15 1.767 super().finalize_options()
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/command/install.py", line 407, in finalize_options
15:49:45 #15 1.767 'dist_fullname': self.distribution.get_fullname(),
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_core_metadata.py", line 266, in get_fullname
15:49:45 #15 1.767 return _distribution_fullname(self.get_name(), self.get_version())
15:49:45 #15 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
15:49:45 #15 1.767 canonicalize_version(version, strip_trailing_zero=False),
15:49:45 #15 1.767 TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
15:49:45 #15 1.767
15:49:45 #15 1.769 Failed to create virtual environment.
15:49:45 #15 ERROR: process "/bin/sh -c pipenv install --python 3.9" did not complete successfully: exit code: 1
15:49:45 ------
15:49:45 > [11/12] RUN pipenv install --python 3.9:
15:49:45 1.767 super().finalize_options()
15:49:45 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/command/install.py", line 407, in finalize_options
15:49:45 1.767 'dist_fullname': self.distribution.get_fullname(),
15:49:45 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_core_metadata.py", line 266, in get_fullname
15:49:45 1.767 return _distribution_fullname(self.get_name(), self.get_version())
15:49:45 1.767 File "/usr/local/lib/python3.9/dist-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
15:49:45 1.767 canonicalize_version(version, strip_trailing_zero=False),
15:49:45 1.767 TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
15:49:45 1.767
15:49:45 1.769 Failed to create virtual environment.

Steps to replicate

Run the following docker script on a t2.micro AWS EC2 instance built using the following AMI:

amzn2-ami-hvm-2.0.20230926.0-x86_64-gp2
ami-0db2efede9f8bb581
Amazon Linux 2 AMI 2.0.20230926.0 x86_64 HVM gp2

###############DOCKER SCRIPT START###########################
############################################################
FROM ubuntu:focal

ENV TZ=Europe/London
DEBIAN_FRONTEND=noninteractive
IS_DOCKER=true
PYTHONPATH=/code

RUN apt-get -y update
&& apt-get -y install
awscli \
ca-certificates
gnupg
gnupg1
gnupg2
libcairo2 \
libcairo2-dev \
libgeos-dev
libpq-dev
libgirepository1.0-dev
libspatialindex-dev
osm2pgsql
python3.9-dev
&& echo -n 'deb [ arch=amd64 ] https://downloads.skewed.de/apt focal main' >> /etc/apt/sources.list
&& apt-key adv --keyserver keys.openpgp.org --recv-key 612DEFB798507F25
&& apt-get -y update \
&& apt-get install python3-graph-tool=2.45 -y

RUN apt-get install -y curl
RUN curl https://raw.githubusercontent.com/pypa/pipenv/master/get-pipenv.py | python3.9

RUN pipenv install --python 3.9

###############DOCKER SCRIPT END#############################
############################################################

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

Tried running this but it didn't work:

18:07:29 #15 [11/12] RUN pipenv --support | xclip
18:07:29 #15 0.561 /bin/sh: 1: xclip: not found
18:07:31 #15 ERROR: process "/bin/sh -c pipenv --support | xclip" did not complete successfully: exit code: 127

[Whylickspittle]

TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
I meet the same issue

[evallesp]

Here's more info: pypa/setuptools#4483

[matteius]

Here's more info: pypa/setuptools#4483

I suggest following the guidance in the setuptools issue as my current understanding is your issue is actually occurring in setuptools/packaging and should not be considered a pipenv bug.

[viktorvorobev]

Hello!
I've faced the same issue and to me, it helped to add the following, before actually creating pipenv environment:

RUN python3 -m pip install --upgrade --no-cache-dir packaging==22.0
RUN python3 -m pipenv sync --dev

Apparently, packaging is the dependency of virtualenv, while virtualenv is the dependency of pipenv.

I'm using Python 3.8 (because of reasons, don't blame me). With this version and pipenv==2024.0.1:

• if using packaging==21.3, pipenv fails with TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
• if using packaging==22.0, everything seems to be working

Your results may vary depending on used versions of python and all other stuff.

Hopefully that helps!

[matteius]

Should pipenv add packaging>=22.0 to our pypoject.toml dependencies? Would that solve the issue @viktorvorobev ?

[viktorvorobev]

@matteius I believe it could, yes, but it is still a mystery to me, why on Earth, the optional dependency of the virtualenv can break pipenv this way. Guess, you should know this better than me :)

Anyway, I'm happy with my workaround for now, so you do you. My Dockerfile is a mess of its own, so there is no harm in a new line, and for some reason I don't face this issue with newer versions of python or without a docker.

Maybe you should just keep this issue open for the time being, and observe the impact, I don't know ¯\_(ツ)_/¯