test_custom_shebang fails with system (Gentoo) install of PyPy3

[mgorny]

Describe the bug

$ tox -e pypy3
.pkg: _optional_hooks> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_wheel> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
pypy3: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/distlib/.tox/.tmp/package/5/distlib-0.3.8.dev0.tar.gz
pypy3: commands[0]> python tests/test_all.py
............................s.................................................s....s.......s..........................sss.ss.....F....sss......ss.........................s..........s..........................ssss.........ss.............
======================================================================
FAIL: test_custom_shebang (test_scripts.ScriptTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/distlib/tests/test_scripts.py", line 121, in test_custom_shebang
    self.assertEqual(p.returncode, 0)
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 236 tests in 18.396s

FAILED (failures=1, skipped=22)
pypy3: exit 1 (19.16 seconds) /tmp/distlib> python tests/test_all.py pid=237205
.pkg: _exit> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
  pypy3: FAIL code 1 (23.76=setup[4.61]+cmd[19.16] seconds)
  evaluation failed :( (23.81 seconds)

To Reproduce

tox -e pypy3

Expected behavior
Tests passing ;-).

Environment

• OS, including version: Gentoo Linux
• Version of this library: 0.3.7, 445bbf0

Additional information
stdout contains: b"ModuleNotFoundError: No module named 'encodings'\ndebug: OperationError:\ndebug: operror-type: ModuleNotFoundError\ndebug: operror-value: No module named 'encodings'\n"

FWICS this is because you're creating a venv without passing symlinks=True, and venv copies incomplete (nonfunctional) PyPy executable there. That said, I have no clue why the venv module has different defaults than python -m venv ... invocation.

[vsajip]

Which version of PyPy3 are you using? The tests via GitHub Actions are passing with pypy-3.9.

[mgorny]

Which version of PyPy3 are you using? The tests via GitHub Actions are passing with pypy-3.9.

7.3.12. It only happens if PyPy is installed system-wide. When it's run from an isolated tree, it happily copies everything included.

[vsajip]

Could it be distro-related? For example, venv is unbundled from Python in e.g. Ubuntu - you have to install the python3-venv package separately. Not sure if Gentoo packagers have changed any defaults.

[vsajip]

The stdlib venv.create() has a default of symlinks=False, which would apply in the invocation from distlib which just calls venv.create(the_dir). What makes the copied pypy3 interpreter non-functional? It sounds like the fix for this needs to be in pypy.

[mgorny]

Well, there definitely is some regression in PyPy itself because this used to work somehow. Nevertheless, using copy approach is a very bad idea anyway because it means venv copies the whole /usr/lib, and that can easily take a few gigabytes. This is, err, a design choice to make shared linking work with standalone distributions of PyPy. It works when you have a few libraries shipped along with PyPy, it doesn't when PyPy is installed system-wide.

[vsajip]

it means venv copies the whole /usr/lib

Really? Not just /usr/lib/pythonX.Y?

[mgorny]

it means venv copies the whole /usr/lib

Really? Not just /usr/lib/pythonX.Y?

Yes, PyPy vendors a bunch of system shared libraries for portability, so it needs to copy them to venv. However, this has a side effect that when it's installed into the system it doesn't discriminate which libraries it actually needs.

[vsajip]

What happens on Windows? Symlinks don't work on all versions, so copying is usually used on Windows.

[mgorny]

Sorry, I have no clue about Windows. There's no Gentoo/Windows ;-).

[mgorny]

That said, I suppose there's no "system-wide" install to consider on Windows in the first place. I think the cleanest solution for the time being would be to replicate the "copy on Windows, symlinks elsewhere" logic from python -m venv.

[vsajip]

Perhaps, but it seems like PyPy needs to address some issues on their side ...

[mgorny]

Sure, and they're working on it. However, it's going to take quite some time before it gets done and deployed.

[mgorny]

Thanks a lot!