Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pex resolver API is not completely hermetic to PYTHONPATH. #892

Closed
jsirois opened this issue Feb 19, 2020 · 0 comments · Fixed by #895
Closed

Pex resolver API is not completely hermetic to PYTHONPATH. #892

jsirois opened this issue Feb 19, 2020 · 0 comments · Fixed by #895
Assignees

Comments

@jsirois
Copy link
Member

jsirois commented Feb 19, 2020

The pex.resolver module delegates work to pip by running a pip PEX in a subprocess. If the interpreter being resolved for is not the current interpreter and the current interpreter has portions of its stdlib exported on the PYTHONPATH and those portions of the stdlib are incompatible with the interpreter being resolved for, the bootstrap of the pip PEX will fail before it can scrub away the PYTHONPATH.

A minimal repro:

$ python3.8 <<EOF
> import os
> import sys
> 
> 
> # This puts python3.8 stdlib on PYTHONPATH.
> os.environ['PYTHONPATH'] = os.pathsep.join(sys.path)
> 
> 
> from pex import resolver
> from pex.interpreter import PythonInterpreter
> 
> 
> def find_python36():
>   for interp in PythonInterpreter.iter():
>     if interp.version[:2] == (3, 6):
>       return interp
>   raise RuntimeError('This test requires python3.6 be on the PATH.')
> 
> 
> result = resolver.resolve(requirements=['packaging==19.2'], interpreter=find_python36())
> print('Resolved: {}'.format(result))
> EOF
Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
  File "/usr/lib/python3.8/abc.py", line 64, in <module>
ModuleNotFoundError: No module named '_abc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/io.py", line 52, in <module>
  File "/usr/lib/python3.8/abc.py", line 68, in <module>
  File "/usr/lib/python3.8/_py_abc.py", line 35
    def __new__(mcls, name, bases, namespace, /, **kwargs):
                                              ^
SyntaxError: invalid syntax
Traceback (most recent call last):
  File "<stdin>", line 20, in <module>
  File "/home/jsirois/dev/pantsbuild/pex/pex/resolver.py", line 701, in resolve
    return list(resolve_request.resolve_distributions(ignore_errors=ignore_errors))
  File "/home/jsirois/dev/pantsbuild/pex/pex/resolver.py", line 498, in resolve_distributions
    for resolve_result in self._run_parallel(inputs=to_resolve,
  File "/home/jsirois/dev/pantsbuild/pex/pex/resolver.py", line 366, in _run_parallel
    for result in execute_parallel(
  File "/home/jsirois/dev/pantsbuild/pex/pex/jobs.py", line 219, in execute_parallel
    raise error
pex.resolver.Unsatisfiable: pid: 13111 -> /usr/bin/python3.6 /tmp/tmp6ghsjpkj --disable-pip-version-check --isolated --exists-action i -q --no-cache-dir download --dest /tmp/tmpk4po5m_6/resolved_dists/cp36-cp36m packaging==19.2 raised Executing /usr/bin/python3.6 /tmp/tmp6ghsjpkj --disable-pip-version-check --isolated --exists-action i -q --no-cache-dir download --dest /tmp/tmpk4po5m_6/resolved_dists/cp36-cp36m packaging==19.2 failed with -6
@jsirois jsirois self-assigned this Feb 19, 2020
jsirois added a commit to jsirois/pex that referenced this issue Feb 19, 2020
As described in pex-tool#892, when the Pex resolver API is used in-process, a
sufficiently antagonistic PYTHONPATH could lead to a failure to launch
the pip PEX we delegate resolving to. Fix this by scrubbing PYTHONPATH
from the pip PEX environment pre-launch.

Fixes pex-tool#892
jsirois added a commit to jsirois/pex that referenced this issue Feb 20, 2020
As described in pex-tool#892, when the Pex resolver API is used in-process, a
sufficiently antagonistic PYTHONPATH could lead to a failure to launch
the pip PEX we delegate resolving to. Fix this by scrubbing PYTHONPATH
from the pip PEX environment pre-launch.

Fixes pex-tool#892
jsirois added a commit that referenced this issue Feb 20, 2020
As described in #892, when the Pex resolver API is used in-process, a
sufficiently antagonistic PYTHONPATH could lead to a failure to launch
the pip PEX we delegate resolving to. Fix this by scrubbing PYTHONPATH
from the pip PEX environment pre-launch.

Fixes #892
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant