tox -epy39 -- -vvsk test_issues_892 broken on main

[jsirois]

Lots of heat on this one but little light. This is green in CI and used to work for me locally but now does not:

$ tox -epy39 -- -vvsk test_issues_892
py39 inst-nodeps: /home/jsirois/dev/pantsbuild/pex/.tox/.tmp/package/1/pex-2.1.50.tar.gz
py39 installed: atomicwrites==1.4.0,attrs==21.2.0,more-itertools==8.9.0,packaging==21.0,pex @ file:///home/jsirois/dev/pantsbuild/pex/.tox/.tmp/package/1/pex-2.1.50.tar.gz,pkginfo==1.7.0,pluggy==0.13.1,py==1.10.0,pyparsing==2.4.7,pytest==4.6.11,six==1.16.0,wcwidth==0.2.5
py39 run-test-pre: PYTHONHASHSEED='1469193847'
py39 run-test: commands[0] | pytest --ignore=tests/integration -vvsk test_issues_892
==================================================================================================================================== test session starts =====================================================================================================================================
platform linux -- Python 3.9.7, pytest-4.6.11, py-1.10.0, pluggy-0.13.1 -- /home/jsirois/dev/pantsbuild/pex/.tox/py39/bin/python
cachedir: .tox/py39/.pytest_cache
rootdir: /home/jsirois/dev/pantsbuild/pex
collected 319 items / 318 deselected / 1 selected                                                                                                                                                                                                                                            

tests/test_resolver.py::test_issues_892 FAILED

========================================================================================================================================== FAILURES ==========================================================================================================================================
______________________________________________________________________________________________________________________________________ test_issues_892 _______________________________________________________________________________________________________________________________________

    def test_issues_892():
        # type: () -> None
        python27 = ensure_python_interpreter(PY27)
        program = dedent(
            """\
            from __future__ import print_function
    
            import os
            import sys
    
    
            # This puts python3.8 stdlib on PYTHONPATH.
            os.environ['PYTHONPATH'] = os.pathsep.join(sys.path)
    
    
            from pex import resolver
            from pex.interpreter import PythonInterpreter
    
    
            python27 = PythonInterpreter.from_binary({python27!r})
            result = resolver.resolve(requirements=['packaging==19.2'], interpreters=[python27])
            print('Resolved: {{}}'.format(result))
      """.format(
                python27=python27
            )
        )
    
        python38 = ensure_python_interpreter(PY38)
        cmd, process = PythonInterpreter.from_binary(python38).open_process(
            args=["-c", program], stderr=subprocess.PIPE
        )
        _, stderr = process.communicate()
>       assert process.returncode == 0, dedent(
            """
            Command {cmd} failed with {returncode}.
    
            STDERR
            ======
            {stderr}
            """.format(
                cmd=cmd, returncode=process.returncode, stderr=stderr.decode("utf8")
            )
        )
E       AssertionError: 
E                 Command ['/home/jsirois/.pex_dev/pyenv/versions/3.8.11/bin/python3.8', '-s', '-E', '-c', "from __future__ import print_function\n\nimport os\nimport sys\n\n\n# This puts python3.8 stdlib on PYTHONPATH.\nos.environ['PYTHONPATH'] = os.pathsep.join(sys.path)\n\n\nfrom pex import resolver\nfrom pex.interpreter import PythonInterpreter\n\n\npython27 = PythonInterpreter.from_binary('/home/jsirois/.pex_dev/pyenv/versions/2.7.18/bin/python2.7')\nresult = resolver.resolve(requirements=['packaging==19.2'], interpreters=[python27])\nprint('Resolved: {}'.format(result))\n"] failed with 1.
E         
E                 STDERR
E                 ======
E                 Traceback (most recent call last):
E           File "<string>", line 16, in <module>
E           File "/home/jsirois/dev/pantsbuild/pex/pex/resolver.py", line 927, in resolve
E             build_requests, download_results = _download_internal(
E           File "/home/jsirois/dev/pantsbuild/pex/pex/resolver.py", line 1048, in _download_internal
E             download_results = download_request.download_distributions(
E           File "/home/jsirois/dev/pantsbuild/pex/pex/resolver.py", line 125, in download_distributions
E             return list(
E           File "/home/jsirois/dev/pantsbuild/pex/pex/jobs.py", line 428, in execute_parallel
E             raise error
E           File "/home/jsirois/dev/pantsbuild/pex/pex/jobs.py", line 434, in execute_parallel
E             result = error_handler.handle_spawn_error(spawn_result.item, spawn_result.error)
E           File "/home/jsirois/dev/pantsbuild/pex/pex/jobs.py", line 331, in handle_spawn_error
E             raise self._raise_type(self.spawn_error_message(item, exception))
E         pex.resolver.Unsatisfiable: Failed to spawn a job for DistributionTarget(interpreter=PythonInterpreter('/home/jsirois/.pex_dev/pyenv/versions/2.7.18/bin/python2.7', PythonIdentity('/home/jsirois/.pex_dev/pyenv/versions/2.7.18/bin/python2.7', 'cp27', 'cp27mu', 'manylinux_2_33_x86_64', (2, 7, 18)))): received exit code 100 during execution of `['/home/jsirois/.pex_dev/pyenv/versions/2.7.18/bin/python2.7', '-s', '-E', '/tmp/tmpg73ewdk7', '--no-pip', '--no-setuptools', '--no-wheel', '/home/jsirois/.pex/venvs/f908e56bebfa779d2a5f30cde89d4709d1e406f1/73a306939efaa24e3a78f6cd6354557b299a45f3.3766c795a4864563aed4ba8c58ea22ae']` while trying to execute `['/home/jsirois/.pex_dev/pyenv/versions/2.7.18/bin/python2.7', '-s', '-E', '/tmp/tmpg73ewdk7', '--no-pip', '--no-setuptools', '--no-wheel', '/home/jsirois/.pex/venvs/f908e56bebfa779d2a5f30cde89d4709d1e406f1/73a306939efaa24e3a78f6cd6354557b299a45f3.3766c795a4864563aed4ba8c58ea22ae']`
E         
E         
E       assert 1 == 0
E         -1
E         +0

tests/test_resolver.py:411: AssertionError
========================================================================================================================== 1 failed, 318 deselected in 2.00 seconds ==========================================================================================================================
ERROR: InvocationError for command /home/jsirois/dev/pantsbuild/pex/.tox/py39/bin/pytest --ignore=tests/integration -vvsk test_issues_892 (exited with code 1)
__________________________________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________________________________
ERROR:   py39: commands failed

It turns out this is because, despite Pex creating a venv using -E, the virtualenv.py vendored in to create Python 2.7 and PyPy venvs appears to read PYTHONPATH on its own and choke:

$ PYTHONPATH=/home/jsirois/.pyenv/versions/3.8.11/lib/python3.8 /home/jsirois/.pex_dev/pyenv/versions/2.7.18/bin/python2.7 -sE pex/tools/commands/virtualenv_16.7.12_py --no-pip --no-setuptools --no-wheel /tmp/bob
New python executable in /tmp/bob/bin/python2.7
Also creating executable in /tmp/bob/bin/python
  File "/home/jsirois/.pyenv/versions/3.8.11/lib/python3.8/site.py", line 178
    file=sys.stderr)
        ^
SyntaxError: invalid syntax
ERROR: The executable /tmp/bob/bin/python2.7 is not functioning
ERROR: It thinks sys.prefix is '/home/jsirois/dev/pantsbuild/jsirois-pex' (should be u'/tmp/bob')
ERROR: virtualenv is not compatible with this system or executable

That's vs:

$ PYTHONPATH=/home/jsirois/.pyenv/versions/3.8.11/lib/python3.8 /home/jsirois/.pex_dev/pyenv/versions/2.7.18/bin/python2.7 -s -c 'print("OK")'
  File "/home/jsirois/.pyenv/versions/3.8.11/lib/python3.8/site.py", line 178
    file=sys.stderr)
        ^
SyntaxError: invalid syntax
$ PYTHONPATH=/home/jsirois/.pyenv/versions/3.8.11/lib/python3.8 /home/jsirois/.pex_dev/pyenv/versions/2.7.18/bin/python2.7 -sE -c 'print("OK")'
OK

It looks like something like the fix for #892 in #895 will need to be applied to virtualenv creation.

[Eric-Arellano]

If Pex no longer supported Python 2, it sounds like this issue wouldn't be the case? Or at least easier to fix.

If so, that would be the second Pex issue you filed this week where our life is harder because of python 2. (Other is the error for conflicting constraints file). I continue to encourage removing python 2 support in the near future.

[jsirois]

I know you do ;)

[jsirois]

And "our" life is not harder in a burdensome way. At least not mine. I'm happy to maintain long term compatibility for users. Full stop.

Pex will be moving on in order to unblock a Pip upgrade that brings fixes - but only for that reason.