test_bdb and test_distutils fail on installed Python 3.9, 3.10 and 3.11 (setuptools 60.9.3, pip 22.0.4)

[vstinner]

BPO	47013
Nosy	@jaraco, @vstinner, @ned-deily

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-03-14.16:14:07.532>
labels = ['3.8', '3.9', '3.10', '3.11', '3.7', 'tests']
title = 'test_bdb and test_distutils fail on installed Python 3.9, 3.10 and 3.11 (setuptools 60.9.3, pip 22.0.4)'
updated_at = <Date 2022-03-16.01:15:39.379>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-03-16.01:15:39.379>
actor = 'jaraco'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2022-03-14.16:14:07.532>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 47013
keywords = []
message_count = 8.0
messages = ['415157', '415158', '415160', '415161', '415162', '415164', '415180', '415311']
nosy_count = 3.0
nosy_names = ['jaraco', 'vstinner', 'ned.deily']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue47013'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

[vstinner]

Since build 298, test_bdb test_distutils fail on s390x Fedora Rawhide Clang Installed 3.10:
https://buildbot.python.org/all/#/builders/694/builds/298

2 changes of build 298:

• bpo-46986: Upgrade bundled setuptools to 60.9.3 (bpo-46986: Upgrade bundled setuptools to 60.9.3 #31820)(21 hours ago)
• [3.10] bpo-46985: Upgrade bundled pip to 22.0.4 (bpo-46985: Upgrade bundled pip to 22.0.4 #31819) ([3.10] bpo-46985: Upgrade bundled pip to 22.0.4 (GH-31819) #31849)(21 hours ago)

Logs:

0:02:03 load avg: 9.95 [213/427/1] test_distutils failed (uncaught exception) -- running: test_concurrent_futures (42.4 sec)
test test_distutils crashed -- Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/libregrtest/runtest.py", line 335, in _runtest_inner
    refleak = _runtest_inner2(ns, test_name)
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/libregrtest/runtest.py", line 280, in _runtest_inner2
    the_module = importlib.import_module(abstest)
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_distutils.py", line 15, in <module>
    import distutils.tests
ModuleNotFoundError: No module named 'distutils.tests'

======================================================================
FAIL: test_skip (test.test_bdb.StateTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_bdb.py", line 730, in test_skip
    with TracerRun(self, skip=skip) as tracer:
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_bdb.py", line 448, in __exit__
    self.test_case.fail(err_msg)
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_bdb.py", line 582, in fail
    raise self.failureException(msg) from None
AssertionError: Wrong event type at expect_set item 2, got 'call'
  Expected: ('line', 3, 'tfunc_import')
  Got:      ('call', 84, 'find_spec'),      ('quit',),

[vstinner]

Reproduce:

$ ./configure --prefix /opt/py310 && make clean && make && make install 
$ cd /somewhere/else/
$ /opt/py310/bin/python3 -m test -v test_bdb
...
FAIL: test_skip (test.test_bdb.StateTestCase)
...
$ /opt/py310/bin/python3 -m test -v test_distutils 
...
ModuleNotFoundError: No module named 'distutils.tests'
...

setuptools installs this file:

$ cat /opt/py310/lib/python3.10/site-packages/distutils-precedence.pth
import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'local') == 'local'; enabled and __import__('_distutils_hack').add_shim();

[vstinner]

See: [BUG] Having setuptools installed causes cpython stdlib build to fail
pypa/setuptools#3007

[vstinner]

x86 Gentoo Installed with X 3.9:
https://buildbot.python.org/all/#builders/527/builds/474

4 tests failed:
test_bdb test_distutils test_importlib test_peg_generator

Python 3.9 gets two more issues: test_importlib and test_peg_generator.

======================================================================
FAIL: test_package_discovery (test.test_importlib.test_main.DiscoveryTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/test/test_importlib/test_main.py", line 159, in test_package_discovery
    assert all(
AssertionError

0:06:27 load avg: 3.96 [ 83/425/3] test_peg_generator failed (uncaught exception) -- running: test_multiprocessing_forkserver (1 min 27 sec)
Warning -- warnings.filters was modified by test_peg_generator
Failed to import test module: test.test_peg_generator.test_c_parser
Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/test/test_peg_generator/test_c_parser.py", line 4, in <module>
    from distutils.tests.support import TempdirManager
ModuleNotFoundError: No module named 'distutils.tests'

[vstinner]

AMD64 Fedora Stable Clang Installed 3.x:
https://buildbot.python.org/all/#builders/350/builds/1621

2 re-run tests:
test_bdb test_distutils

[vstinner]

$ ./configure --prefix /opt/py310 && make clean && make && make install
$ cd /somewhere/else/
$ /opt/py310/bin/python3 -m test -v test_bdb

I also reproduce test_bdb and test_distutils failures on the main branch.

[ned-deily]

(The buildbot failures were triggered by yesterday's merges for bpo-46986 which updated the bundled setuptools in ensurepip. I'm reverting those merges now to unblock planned releases.)

[jaraco]

See pypa/setuptools#3007 (comment) where I did a brief analysis and probable explanation.

What it boils down to: Setuptools needs to supply its own copy of distutils aggressively. It provides an opt out for this behavior and is able to offer built in opt out based on whatever signal is available. The default opt out is the SETUPTOOLS_USE_DISTUTILS=stdlib env var. for CPython, it also disables the behavior when ./pybuilddir.txt is present, but installed Pythons probably don’t have this config.

Can those test runners be updated to set this environment variable?

[jaraco]

From pypa/setuptools#3007:

If there is a pybuilddir.txt next to setup.py then it's a CPython Unix build. All the other factors are not reliable.

If tiran is right, then Setuptools is already doing everything it can not to interfere with CPython builds.

SETUPTOOLS_USE_DISTUTILS is not an option. We cannot reliable inject the env var into our tests harness. The problem has to be solved in setuptools.

It appears that Setuptools is unable to do anything to detect that the stdlib distutils is to be preferred.

Trying to think outside the box, maybe there's a way that the distutils hook could be extended to detect the import of any module matching test.test_, and if that's the case, disable the hook as well.

It's not simply enough to detect an attempt to import test, because test.support is available and used by third party packages.

In fact, maybe Setuptools could simply disable the hook if the specifically-implicated modules are encountered.

I can try that, maybe using the repro above.

[jaraco]

I was able to repro at least some of the failures with this dockerfile:

FROM jaraco/multipy-tox

RUN apt install -y libarchive-tools libssl-dev
RUN wget -O - https://github.com/python/cpython/archive/refs/heads/3.9.zip | bsdtar -xf -
WORKDIR cpython-3.9
RUN sh ./configure --prefix /opt/python
RUN make install
WORKDIR /
RUN /opt/python/bin/python3 -m pip install --upgrade pip setuptools
CMD /opt/python/bin/python3 -m test

Here's an excerpt from the output:

5 tests failed:
    test_bdb test_distutils test_importlib test_peg_generator
    test_zipfile

[jaraco]

Attempting to install a pre-release of setuptools, I find that the custom build of Python is inadequate because _ctypes isn't available, so import wheel.macosx_libfile fails. I need to figure out what dependencies are needed to get ctypes to build.

I suspect libffi-dev was the required dependency, but I've just copied all of the dependencies from the developer guide:

FROM jaraco/multipy-tox

RUN apt install -y libarchive-tools
RUN apt install -y build-essential gdb lcov pkg-config libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev
RUN wget -O - https://github.com/python/cpython/archive/refs/heads/3.9.zip | bsdtar -xf -
WORKDIR cpython-3.9
RUN sh ./configure --prefix /opt/python
RUN make install
WORKDIR /
RUN /opt/python/bin/python3 -m pip install --upgrade pip
RUN /opt/python/bin/python3 -m pip install git+https://github.com/pypa/setuptools@experiment/sensitive-tests
CMD /opt/python/bin/python3 -m test

[jaraco]

My attempts to have setuptools selectively disable the distutils override worked for all of the tests except test_bdb. The failing test is test_skip. I suspected and then confirmed that that test fails even if the distutils patching is disabled but the Finder is still present. It seems the presence of another finder on meta_path is sufficient to break test_bdb.StateTestCase.test_skip.

[jaraco]

Here's a repro not involving Setuptools showing how test_bdb is sensitive to sys.meta_path manipulation:

FROM jaraco/multipy-tox

RUN apt install -y libarchive-tools
RUN apt install -y build-essential gdb lcov pkg-config libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev
RUN wget -O - https://github.com/python/cpython/archive/refs/heads/3.9.zip | bsdtar -xf -
WORKDIR cpython-3.9
RUN sh ./configure --prefix /opt/python
RUN make install
WORKDIR /
RUN /opt/python/bin/python3 -m pip install --upgrade pip
RUN /opt/python/bin/python3 -m pip uninstall -y setuptools
RUN echo "import sys; sys.meta_path.insert(0, type('NullFinder', (), dict(find_spec=lambda *args: None))())" > /opt/python/lib/python3.9/site-packages/null-finder.pth
CMD /opt/python/bin/python3 -m test_bdb

It fails with a similar error:

AssertionError: Wrong event type at expect_set item 2, got 'call'
  Expected: ('line', 3, 'tfunc_import')
  Got:      ('call', 1, '<lambda>'),        ('quit',),

[jaraco]

Based on this finding, I don't think there's anything that Setuptools can do short of completely removing the hook, which is a non-starter because (a) it would break all users relying on the hook to provide distutils when it's not available in the stdlib and (b) it would remove the ability for use cases still reliant on stdlib to opt out.

I suggest we'll want to find a fix for test_bdb independent of the concerns introduced by setuptools.

[jaraco]

With this Dockerfile, I can confirm the proposed fix in Setuptools has the desired effect:

FROM jaraco/multipy-tox

RUN apt install -y libarchive-tools
RUN apt install -y build-essential gdb lcov pkg-config libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev
RUN wget -O - https://github.com/python/cpython/archive/refs/heads/3.9.zip | bsdtar -xf -
WORKDIR cpython-3.9
RUN sh ./configure --prefix /opt/python
RUN make install
RUN chmod u+x /opt/python/lib/python3.9/test/ziptestdata/exe_with_*
WORKDIR /
RUN /opt/python/bin/python3 -m pip install --upgrade pip
RUN /opt/python/bin/python3 -m pip install git+https://github.com/pypa/setuptools@bugfix/3383-cpython-sensitive-tests
CMD /opt/python/bin/python3 -m test

$ docker run -it -v /Users/jaraco/p/pypa/setuptools:/src @$(docker build -q .) /opt/python/bin/python3 -m test test_importlib test_distutils test_peg_generator
0:00:00 load avg: 0.11 Run tests sequentially
0:00:00 load avg: 0.11 [1/3] test_importlib
0:00:01 load avg: 0.11 [2/3] test_distutils
/opt/python/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
0:00:02 load avg: 0.11 [3/3] test_peg_generator
/opt/python/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

== Tests result: SUCCESS ==

All 3 tests OK.

Total duration: 3.0 sec
Tests result: SUCCESS

Unfortunately, it also creates its own regression in test_venv:

$ docker run -it -v /Users/jaraco/p/pypa/setuptools:/src @$(docker build -q .) /opt/python/bin/py
thon3 -m test test_venv
0:00:00 load avg: 0.04 Run tests sequentially
0:00:00 load avg: 0.04 [1/1] test_venv
test test_venv failed -- Traceback (most recent call last):
  File "/opt/python/lib/python3.9/test/test_venv.py", line 544, in test_with_pip
    self.do_test_with_pip(True)
  File "/opt/python/lib/python3.9/test/test_venv.py", line 511, in do_test_with_pip
    out, err = check_output([envpy,
  File "/opt/python/lib/python3.9/test/test_venv.py", line 44, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['/tmp/tmpqts0cln1/bin/python3', '-W', 'ignore::DeprecationWarning', '-I', '-m', 'ensurepip._uninstall']' returned non-zero exit status 1.

test_venv failed (1 error)

== Tests result: FAILURE ==

1 test failed:
    test_venv

Total duration: 5.7 sec
Tests result: FAILURE

And annoyingly, that error message is all but worthless for diagnosis.

[jaraco]

After adding a breakpoint where the error is occurring (pytest would do this automatically with --pdb), I'm able to inspect the error:

(Pdb) print(exc.stderr.decode())
Traceback (most recent call last):
  File "<string>", line 6, in <module>
  File "/opt/python/lib/python3.9/runpy.py", line 221, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/opt/python/lib/python3.9/runpy.py", line 130, in _get_module_details
    spec = importlib.util.find_spec(mod_name)
  File "/opt/python/lib/python3.9/importlib/util.py", line 103, in find_spec
    return _find_spec(fullname, parent_path)
  File "<frozen importlib._bootstrap>", line 925, in _find_spec
  File "/tmp/tmpntevhrty/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 79, in find_spec
    return method()
  File "/tmp/tmpntevhrty/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 100, in spec_for_pip
    if self.pip_imported_during_build():
  File "/tmp/tmpntevhrty/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 111, in pip_imported_during_build
    return any(
  File "/tmp/tmpntevhrty/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 112, in <genexpr>
    frame.f_globals['__file__'].endswith('setup.py')
KeyError: '__file__'
Traceback (most recent call last):
  File "/opt/python/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/python/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/python/lib/python3.9/ensurepip/_uninstall.py", line 31, in <module>
    sys.exit(_main())
  File "/opt/python/lib/python3.9/ensurepip/_uninstall.py", line 27, in _main
    return ensurepip._uninstall_helper(verbosity=args.verbosity)
  File "/opt/python/lib/python3.9/ensurepip/__init__.py", line 152, in _uninstall_helper
    return _run_pip(args + [p[0] for p in reversed(_PROJECTS)])
  File "/opt/python/lib/python3.9/ensurepip/__init__.py", line 34, in _run_pip
    return subprocess.run([sys.executable, "-c", code], check=True).returncode
  File "/opt/python/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/tmpntevhrty/bin/python3', '-c', '\nimport runpy\nimport sys\nsys.path = [] + sys.path\nsys.argv[1:] = [\'uninstall\', \'-y\', \'--disable-pip-version-check\', \'pip\', \'setuptools\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.

It seems there's a new issue, probably brought on by changes in Setuptools.

[jaraco]

The failure looks suspiciously similar to pypa/setuptools#2940.

[jaraco]

Okay, so Setuptools fixed that issue and released it in 60.0.3, but the version of Setuptools installed into the virtualenv doesn't have that fix. But that doesn't explain why the issue only emerges when the version of Setuptools is bumped.

Aah. I get it. This test is running a venv with --system-site-packages, so the distutils-precedence.pth for the system site packages is taking effect, enabling SETUPTOOLS_USE_DISTUTILS=local, but then because the venv gets an old version of Setuptools with the bug in it, that version gets invoked. This bug will go away when the version of Setuptools installed into the venvs is updated.

[jaraco]

At this point, pypa/setuptools#3386 (to be released with Setuptools>=62.5.0) and gh-93962 should fix the issue.

[yilun11]

@jaraco I upgraded to setuptools 64.0.3 and still get these errors. Is there a solution?

Using docker pull registry.gitlab.com/python-devs/ci-images:active from README for dev container
python3 -m test test_importlib test_distutils test_peg_generator

All 3 tests fail due to no module found. Pip install --upgrade setuptools doesn't resolve although version is 64.0.3. I also tried make from source and it throws the error at test_peg_generator.

Thanks!

[jaraco]

@yilun11 I suspect your issue is different than this one. I can indeed produce an error if I use your repro:

 ~ $ docker run -it registry.gitlab.com/python-devs/ci-images:active python3 -m test test_importlib | grep ModuleNotFound
ModuleNotFoundError: No module named 'test.test_importlib'

However, I also observe that I get the same error for any test module:

 ~ $ docker run -it registry.gitlab.com/python-devs/ci-images:active python3 -m test test_bool | grep ModuleNotFound
ModuleNotFoundError: No module named 'test.test_bool'

When I look at the Python lib directory, I see a support subpackage, but no modules for the tests themselves. I suspect the image you're using simply doesn't have the Python tests installed, so you shouldn't expect it to work. The ModuleNotFoundError is a fairly generic error not specific to this issue. This issue is specific to a few select modules not working when Setuptools is present. You can tell if you're affected by the issue if you can work around it by setting SETUPTOOLS_USE_DISTUTILS=stdlib.

To close the loop on this issue, I've confirmed that the repro that previously failed no longer fails with the latest Setuptools:

 draft $ cat Dockerfile
FROM jaraco/multipy-tox

RUN apt install -y libarchive-tools
RUN apt install -y build-essential gdb lcov pkg-config libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev
RUN wget -O - https://github.com/python/cpython/archive/refs/heads/3.9.zip | bsdtar -xf -
WORKDIR cpython-3.9
RUN sh ./configure --prefix /opt/python
RUN make install
RUN chmod u+x /opt/python/lib/python3.9/test/ziptestdata/exe_with_*
WORKDIR /
RUN /opt/python/bin/python3 -m pip install --upgrade pip setuptools
CMD /opt/python/bin/python3 -m test
 draft $ docker run -it @$(docker build -q .) /opt/python/bin/python3 -m test test_importlib test_distutils test_peg_generator
0:00:00 load avg: 0.63 Run tests sequentially
0:00:00 load avg: 0.63 [1/3] test_importlib
0:00:01 load avg: 0.63 [2/3] test_distutils
/opt/python/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
0:00:02 load avg: 0.66 [3/3] test_peg_generator
/opt/python/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

== Tests result: SUCCESS ==

All 3 tests OK.

Total duration: 2.9 sec
Tests result: SUCCESS