[BUG] error in XXXX setup command: use_2to3 is invalid when false.

[miguelfito]

setuptools version

setuptools==58.0.2

Python version

Python 3.6.9, Python 3.9.2

OS

Ubuntu 18.04.5 LTS, Debian 11

Additional environment information

This happens installing any Python2 package when trying to use use_2to3 on setuptool execution

Description

Error installing a package that requires to use use_2to3 on its installation

Expected behavior

The package should be installed as previous versions do (like setuptools==58.0.1)

How to Reproduce

1. Generate a virtualenv and activate it.
2. Make sure you're environment tools are updated.
3. Install a Python2 package that requires to use use_2to3 on its installation.

Output

1. Generate a virtualenv like:

virtualenv .env -p /usr/bin/python3 && source .env/bin/activate
created virtual environment CPython3.9.2.final.0-64 in 121ms
  creator CPython3Posix(dest=/home/miguelfito/tmp/pruebas/devops_housekeeping_test/.env, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/miguelfito/.local/share/virtualenv)
    added seed packages: pip==20.3.4, pkg_resources==0.0.0, setuptools==44.1.1, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

2. Make sure you're environment tools are updated:

python3 -m pip install --upgrade pip setuptools wheel
Looking in indexes: ****
Requirement already satisfied: pip in ./.env/lib/python3.9/site-packages (20.3.4)
Collecting pip
  Downloading pip-21.2.4-py3-none-any.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 5.4 MB/s 
Requirement already satisfied: setuptools in ./.env/lib/python3.9/site-packages (44.1.1)
Collecting setuptools
  Downloading setuptools-58.0.2-py3-none-any.whl (816 kB)
     |████████████████████████████████| 816 kB 20.9 MB/s 
Requirement already satisfied: wheel in ./.env/lib/python3.9/site-packages (0.34.2)
Collecting wheel
  Downloading wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: wheel
    Found existing installation: wheel 0.34.2
    Uninstalling wheel-0.34.2:
      Successfully uninstalled wheel-0.34.2
  Attempting uninstall: setuptools
    Found existing installation: setuptools 44.1.1
    Uninstalling setuptools-44.1.1:
      Successfully uninstalled setuptools-44.1.1
  Attempting uninstall: pip
    Found existing installation: pip 20.3.4
    Uninstalling pip-20.3.4:
      Successfully uninstalled pip-20.3.4
Successfully installed pip-21.2.4 setuptools-58.0.2 wheel-0.37.0

3. Install a Python2 package:

pip install mailer==0.8.1                            
Looking in indexes: **********
Collecting mailer==0.8.1
  Downloading mailer-0.8.1.zip (6.5 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/miguelfito/tmp/pruebas/devops_housekeeping_test/.env/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-k06h_9aq/mailer_e0ec5902e983461198885f3ed8f7b2db/setup.py'"'"'; __file__='"'"'/tmp/pip-install-k06h_9aq/mailer_e0ec5902e983461198885f3ed8f7b2db/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-y8q_t3za
         cwd: /tmp/pip-install-k06h_9aq/mailer_e0ec5902e983461198885f3ed8f7b2db/
    Complete output (1 lines):
    error in mailer setup command: use_2to3 is invalid.
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/30/e7/c3a932fc03ab95c96ff911d8ee3e89b937205596a5c1f00849401a2468e0/mailer-0.8.1.zip#sha256=3411a12197731e0d5379ab194d9acc8d0d71c8b95fdfb11474d67c3e9860070e (from https://pypi.org/simple/mailer/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Downloading https:// **********/30/e7/c3a932fc03ab95c96ff911d8ee3e89b937205596a5c1f00849401a2468e0/mailer-0.8.1.zip (6.5 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/miguelfito/tmp/pruebas/devops_housekeeping_test/.env/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-k06h_9aq/mailer_ac6269ca2eb7417faf12a97364828523/setup.py'"'"'; __file__='"'"'/tmp/pip-install-k06h_9aq/mailer_ac6269ca2eb7417faf12a97364828523/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-_1878lem
         cwd: /tmp/pip-install-k06h_9aq/mailer_ac6269ca2eb7417faf12a97364828523/
    Complete output (1 lines):
    error in mailer setup command: use_2to3 is invalid.
    ----------------------------------------
WARNING: Discarding https:// **********/30/e7/c3a932fc03ab95c96ff911d8ee3e89b937205596a5c1f00849401a2468e0/mailer-0.8.1.zip#sha256=3411a12197731e0d5379ab194d9acc8d0d71c8b95fdfb11474d67c3e9860070e (from https://descinet.bbva.es/artifactory/api/pypi/pypi/simple/mailer/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement mailer==0.8.1 (from versions: 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8.1)
ERROR: No matching distribution found for mailer==0.8.1

Code of Conduct

• I agree to follow the PSF Code of Conduct

[rawteech]

Thanks for raising the issue @miguelfito

Also good to note that this bug is causing errors in downstream dependancies like FlaskOpenID

[0robustus1]

May i suggest to revert #2770 and push a new patch level release?
It seems that this change should also cause a major version release (since it is breaking for some/many packages that still use use2to3).

[mattclay]

There was no deprecation warning for the change in #2770 for packages that disable the use_2to3 feature.

It doesn't make sense to break installation of packages which disable the feature, without any notice. Even if there was a deprecation warning, disabling the feature probably shouldn't break installation.

[webknjaz]

It sounds like it was assumed that since setuptools no longer supports Python 2 (it was dropped around 21 months ago — https://setuptools.rtfd.io/en/latest/history.html#v45-0-0), nobody would hit this anyway 🤷‍♂️.

[webknjaz]

FWIW I see why this could've been expected to be dropped in a major version. So I'll recommend @jaraco to re-release as v59 and yank v58.0.2 on PyPI.

To all the others, if you legitimately think that this should be resurrected, it'd be useful if somebody could come up with a failing regression test that could be merged as XFAIL per https://pganssle-talks.github.io/xfail-lightning/. I don't have a strong case for having this supported in a post Py2 EOL world, though, and would understand if Jason wouldn't want to recover this, especially since he is the only active maintainer in the whole world (without exaggeration).

[nitzmahone]

Seems like there are some ancient projects down low in common projects' dependency Jenga that actually do use it (another Ansible team just tripped over Celery's indirect use of anyjson, which hasn't seen a release since 2012 and actually still uses it). Not necessarily suggesting that the functionality be preserved, but the sudden fail-fast is definitely rippling through the CI world today, especially in places that can't easily pin a setuptools version (eg dynamic uses of virtualenv that are usually just grabbing the latest).

[webknjaz]

(eg dynamic uses of virtualenv that are usually just grabbing the latest).

Try setting an env var PIP_CONSTRAINTS=a_file_limiting_setuptools_version.txt?

[mattclay]

@webknjaz While I agree yanking 58.0.2 makes sense, I don't think a re-release as 59 does, given the lack of a deprecation warning in some cases.

[jimmyhli]

I think @plumdog 's #2769 was meant to fail builds that use the use_2to3 feature, not the builds that disable the use of use_2to3 feature, is that right?

[plumdog]

@HaoYuanLi yes, that was my thought. Specifically that failing for a package that wanted to use 2to3 was better than installing it "successfully" without applying 2to3, as 58.0.0 and 58.0.1 did.

Edit: so if I've done it right, I think that #2777 results in the behaviour I had in mind.

[plumdog]

Also another thought regarding the initial issue report, and how it relates to my intent when I raised #2769:
mailer==0.8.1 installs with setuptools==58.0.1, but is not importable:

$ python --version && pip install setuptools==58.0.1 && pip install mailer==0.8.1 && python -c 'import mailer'
Python 3.9.6
Collecting setuptools==58.0.1
  Using cached setuptools-58.0.1-py3-none-any.whl (816 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 56.0.0
    Uninstalling setuptools-56.0.0:
      Successfully uninstalled setuptools-56.0.0
Successfully installed setuptools-58.0.1
Collecting mailer==0.8.1
  Using cached mailer-0.8.1.zip (6.5 kB)
Using legacy 'setup.py install' for mailer, since package 'wheel' is not installed.
Installing collected packages: mailer
    Running setup.py install for mailer ... done
Successfully installed mailer-0.8.1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/user/example/venv/lib/python3.9/site-packages/mailer.py", line 36, in <module>
    import Queue
ModuleNotFoundError: No module named 'Queue'

(Queue being a module that got moved in the Python 2->3 stdlib shuffle, this indicating that 2to3 was not applied here)

So I think what has happened here is exactly what I'd hoped when I raised #2769: you found a bug at pip install-time, not at runtime. If you have a dependency that needs 2to3, then 58.0.1 is not doing what you want it to, you need 57.

[jaraco]

There was no deprecation warning for the change in #2770 for packages that disable the use_2to3 feature.

I honestly did not expect that anyone would have specified use2to3=False. The primary concern here is to fail fast for those projects relying on use_2to3=True.

May i suggest to revert #2770 and push a new patch level release?
It seems that this change should also cause a major version release (since it is breaking for some/many packages that still use use2to3).

#2770 is a tweak on the original incompatibility introduced on the 58 release. That's where the main incompatibility was introduced. v58.0.2 simply fails faster and in closer proximity to the cause.

I'll restore compatibility for use_2to3=False and regret the inconvenience caused.

FWIW I see why this could've been expected to be dropped in a major version. So I'll recommend @jaraco to re-release as v59 and yank v58.0.2 on PyPI.

I doubt making a backward-incompatible release will have any benefit. v58.0.3 will restore support for use_2to3=False. If we were to release support for Setuptools 59 as described, I can't imagine any users would pin to 58<setuptools<59.

To all the others, if you legitimately think that this should be resurrected, it'd be useful if somebody could come up with a failing regression test that could be merged as XFAIL per https://pganssle-talks.github.io/xfail-lightning/. I don't have a strong case for having this supported in a post Py2 EOL world, though, and would understand if Jason wouldn't want to recover this, especially since he is the only active maintainer in the whole world (without exaggeration).

The main case to be made is for old, unmaintained code that's still in Python 2 syntax but installed on Python 3. I don't want to support this use case any longer. I did expect there would be some disruption from this change and my hope was that it would be small (a few fringe packages), and thus believe a deprecation period to be possibly unnecessary. I'm even offering to help revive/maintain some of these packages.

[jaraco]

Fixed in #2777.

[jimmyhli]

On the other hand, how will this be handled in future releases of setuptools? For packages that have use_2to3=False

[webknjaz]

On the other hand, how will this be handled in future releases of setuptools? For packages that have use_2to3=False

FTR this manifests itself as #2776.