[BUG] PIP3 Install hangs indefinitely

[mgorsk1]

setuptools version

setuptools==58.0.2

Python version

Python 3.7

OS

CentOS Linux 7 (in Docker)

Additional environment information

Running code in gitlab ci/cd docker runner. Gitlab version 14.0.1.

Description

We run pip3 install --upgrade setuptools pip and after that pip3 install -r requirements-dev.txt with following content:

mock
coverage
flake8
mypy
pytest
apache-airflow==1.10.11
types-PyYAML
types-requests

Until yesterday this step finished in under 2 minutes with setuptools==58.0.0. Today newer version of setuptools has been pulled which made step timeout after 1hr. The job is stuck on pip3 install step with croniter package being consistently the last one (stuck for ~55 minutes) before timeout happens.

Expected behavior

pip3 install -r requirements-dev.txt finishes ok

How to Reproduce

run pip3 install --upgrade setuptools pip and after that pip3 install -r requirements-dev.txt with following content:

mock
coverage
flake8
mypy
pytest
apache-airflow==1.10.11
types-PyYAML
types-requests

Output

  Downloading https://xxx/artifactory/api/pypi/ISP-PyPI/packages/packages/28/27/f2c883373aa58fe77ecfc09a984f1ae52bc05790c0eee66b87f65d14cbfe/croniter-0.3.23.tar.gz (18 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading https://xxx/artifactory/api/pypi/ISP-PyPI/packages/packages/79/f1/3d9dac6f43a1f789ad79c1dc2589e7389fc4f0c4753a451485ecb326c357/croniter-0.3.22.tar.gz (18 kB)
  Downloading https://xxx/artifactory/api/pypi/ISP-PyPI/packages/packages/69/84/1a839fdb454929b2f5f34dfc0f6ae9c393193e4c088fa62b37ec318c0df4/croniter-0.3.20.tar.gz (18 kB)
ERROR: Job failed: execution took longer than 1h0m0s seconds

Code of Conduct

• I agree to follow the PSF Code of Conduct

[webknjaz]

I can see how this may be a side-effect of #2775 — the new pip resolver (resolvelib) supports backtracking. It means that if a build fails, it'll try to build an older version of the dependency. But if there are dists that have use_2to3 and a lot of releases, this combination would cause pip attempt to build every version impacting the whole install session before it fails for good.

It's just a speculation, though.

cc @pradyunsg @jaraco

[webknjaz]

@mgorsk1 please, verify if the new release v58.0.3 fixes your issue. It contains a hotfix from #2777.

[webknjaz]

I tried reproducing this with w/ v58.0.3 in the venv under Python 3.9 and I see that my hunch about pip's backtracking was right. It's attempting to download a huge matrix of versions of transitive dependencies and dismisses some of the sdists because of failing to build them (when those projects don't ship wheels).

One of the errors I saw in the log was a problem with Flask-OpenID (see pallets-eco/flask-openid#60):

    ERROR: Command errored out with exit status 1:
     command: ~/src/experiments/setuptools-v58.0.3/venv-s58.0.3/bin/python3.9 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-6ihoqio_/flask-openid_2d8e823c8686418988ff2368e10ca4ea/setup.py'"'"'; __file__='"'"'/tmp/pip-install-6ihoqio_/flask-openid_2d8e823c8686418988ff2368e10ca4ea/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-utc73wga
         cwd: /tmp/pip-install-6ihoqio_/flask-openid_2d8e823c8686418988ff2368e10ca4ea/
    Complete output (1 lines):
    error in Flask-OpenID setup command: use_2to3 is invalid.
    ----------------------------------------

Another example of a discarded version was not a hard failure but a warning about the metadata mismatch for the dill distribution:

  WARNING: Requested dill<0.4,>=0.2.2 from https://files.pythonhosted.org/packages/3e/ad/31932a4e2804897e6fd2f946d53df51dd9b4aa55e152b5404395d00354d1/dill-0.3.1.tar.gz#sha256=d3ddddf2806a7bc9858b20c02dc174396795545e9d62f243b34481fd26eb3e2c (from apache-airflow==1.10.11), but installing version 0.3.1.dev0
WARNING: Discarding https://files.pythonhosted.org/packages/3e/ad/31932a4e2804897e6fd2f946d53df51dd9b4aa55e152b5404395d00354d1/dill-0.3.1.tar.gz#sha256=d3ddddf2806a7bc9858b20c02dc174396795545e9d62f243b34481fd26eb3e2c (from https://pypi.org/simple/dill/) (requires-python:>=2.6, !=3.0.*). Requested dill<0.4,>=0.2.2 from https://files.pythonhosted.org/packages/3e/ad/31932a4e2804897e6fd2f946d53df51dd9b4aa55e152b5404395d00354d1/dill-0.3.1.tar.gz#sha256=d3ddddf2806a7bc9858b20c02dc174396795545e9d62f243b34481fd26eb3e2c (from apache-airflow==1.10.11) has inconsistent version: filename has '0.3.1', but metadata has '0.3.1.dev0'

I've also confirmed that with setuptools < 58 the installation succeeds.

[mgorsk1]

yep even 58.0.0 works just fine but I saw no improvement with 58.0.3

[MichaelStadtmiller]

I am getting an error installing pyhash. It is using setup from setuptools and passing parameter use_2to3=True.
The error was that use_2to3 does not exist for setup. Was this parameter deprecated in 58.0.1/2/3? It is working in 58.0.0.

[jaraco]

yep even 58.0.0 works just fine but I saw no improvement with 58.0.3

The reason 58.0.0 worked fine was because it allowed the project to build and install even though the code was likely invalid.

[jaraco]

Oh, interesting. pyhash has very little Python and running lib2to3 on it produces a very small diff:

pyhash master $ python -m lib2to3 --write *.py
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
--- __init__.py (original)
+++ __init__.py (refactored)
@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals
+
 
 import sys
 import inspect
RefactoringTool: Refactored __init__.py
RefactoringTool: Files that were modified:
RefactoringTool: __init__.py

The easiest fix for this would be for pyhash to release a Python 3 wheel in PyPI of the latest release. It should also consider dropping use of 2to3 as it's not meaningful in that project.

[jaraco]

I filed flier/pyfasthash#49. I don't believe there's anything more to do here. Sorry for the inconvenience.

[potiuk]

Hey @mgorsk1 (and everyone here) -> we solved the issue yesterday for Airflow 2. Together with @jaraco we are now both maintainers of Flask-OpenID, we released https://pypi.org/project/Flask-OpenID/1.3.0/ and updated all the constraints for all released Airflow 2 versions.

Unfortunately Airflow 1.10 is not maintained any more, so we will not update constraints there.

However, there is an easy solution if you are still using using Airflow 1.10 - just copy the constraints to your own git repository and make the change there. The change is super-easy, just change the constraint for Flask-OpenID to 1.3.0 (from 1.2.5).

When you use your own constraints file, this will fix installation problems for you. More you actually won't even need setuptools there for FlaskOpenID, because we relased .whl version of it.

Example commit (this one switches from our tactical solution which implemented before I became a maintainers of Flask OpenID). apache/airflow@be590f4

Big thanks to @mitsuhiko for quick reaction and willingness to help us !

[mgorsk1]

Thanks @potiuk for info, indeed we will just bump airflow dep as we are at 2.1.3 anyways and looks like we just forgot to update unit tests deps 😅