Top level environment markers are lost in built PEXes. #899

jsirois · 2020-02-20T01:05:28Z

When any requirements passed to the Pex CLI have environment markers, these should be recorded in the resulting PEX / PEX-INFO but currently are not. This can lead to erroneous runtime activation failure:

$ python -mpex --python=python2.7 --python=python3.6 "subprocess32==3.2.7; python_version<'3'" -o maybe_subprocess32.pex
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
$ ./maybe_subprocess32.pex 
Python 2.7.17 (default, Oct 22 2019, 09:14:09) 
[GCC 9.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> 
$ python3.6 ./maybe_subprocess32.pex 
Failed to execute PEX file. Needed manylinux2014_x86_64-cp-36-cp36m compatible dependencies for:
 1: subprocess32==3.2.7
    But this pex only contains:
      subprocess32-3.2.7-cp27-cp27mu-linux_x86_64.whl
$ unzip -qc maybe_subprocess32.pex PEX-INFO | jq .
{
  "always_write_cache": false,
  "build_properties": {
    "class": "CPython",
    "pex_version": "2.1.3",
    "platform": "manylinux2014_x86_64",
    "version": [
      2,
      7,
      17
    ]
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "subprocess32-3.2.7-cp27-cp27mu-linux_x86_64.whl": "9c46d1bcfb13ee6660cbd282edbf46be7e941e1f"
  },
  "emit_warnings": true,
  "ignore_errors": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_path": null,
  "requirements": [
    "subprocess32==3.2.7"
  ],
  "zip_safe": true
}

Similar to #898

The text was updated successfully, but these errors were encountered:

jsirois · 2020-02-20T01:08:40Z

Fixing this bug will require parsing requirements. In the switch to pip we ~dropped this and started just passing the requirements through to pip as strings.

jsirois · 2020-02-20T01:22:35Z

The full requirements file spec: https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format
And the implementation: https://github.com/pypa/pip/blob/master/src/pip/_internal/req/req_file.py

jhermann · 2020-02-20T11:36:56Z

https://github.com/sarugaku/requirementslib might help

jsirois · 2020-02-26T05:25:36Z

@jhermann requirementslib would be good but it has fairly out of control transitive dependencies for what Pex needs - just requirements.txt parsing:

$ pip wheel -w requirementslib requirementslib
$ du -sh requirementslib/
3.4M	requirementslib/
$ ls -1 requirementslib/
appdirs-1.4.3-py2.py3-none-any.whl
attrs-19.3.0-py2.py3-none-any.whl
cached_property-1.5.1-py2.py3-none-any.whl
Cerberus-1.3.2-py3-none-any.whl
certifi-2019.11.28-py2.py3-none-any.whl
chardet-3.0.4-py2.py3-none-any.whl
colorama-0.4.3-py2.py3-none-any.whl
distlib-0.3.0-py3-none-any.whl
first-2.0.2-py2.py3-none-any.whl
idna-2.9-py2.py3-none-any.whl
orderedmultidict-1.0.1-py2.py3-none-any.whl
packaging-20.1-py2.py3-none-any.whl
pep517-0.8.1-py2.py3-none-any.whl
pip-20.0.2-py2.py3-none-any.whl
pip_shims-0.5.0-py2.py3-none-any.whl
plette-0.2.3-py2.py3-none-any.whl
pyparsing-2.4.6-py2.py3-none-any.whl
requests-2.23.0-py2.py3-none-any.whl
requirementslib-1.5.3-py2.py3-none-any.whl
setuptools-45.2.0-py3-none-any.whl
six-1.14.0-py2.py3-none-any.whl
toml-0.10.0-py2.py3-none-any.whl
tomlkit-0.5.8-py2.py3-none-any.whl
urllib3-1.25.8-py2.py3-none-any.whl
vistir-0.5.0-py2.py3-none-any.whl
wheel-0.34.2-py2.py3-none-any.whl

I think rolling our own might be the best of bad choices here.

jhermann · 2020-02-26T09:40:42Z

I'd ask them if they're willing to spin off a helper library before that. I found other projects with more of a library character, but they were unmaintained – having a current parsing package would be useful for many.

jsirois · 2020-03-23T19:03:32Z

I stumbled across pip-api which is a good deal slimmer: https://github.com/di/pip-api

This is a prerequisite for fixing #pex-tool#899, implementing pex-tool#1108 and the API will also be useful to Pants which currently can only handle a subset of requirements, forcing edits of existing requirment files to use alternative requirement forms.

This is a prerequisite for fixing ##899, implementing #1108 and the API will also be useful to Pants which currently can only handle a subset of requirements, forcing edits of existing requirment files to use alternative requirement forms.

Work towards pex-tool#899, pex-tool#1020 and pex-tool#1108.

Work towards #899, #1020 and #1108.

This removes our dependency on pkg_resources Environment / WorkingSet in favor of performing our own recursive resolve of runtime distributions to activate using distribution metadata. This fixes an old test bug noticed by Benjy but, more importanty, sets the stage to fix pex-tool#899, pex-tool#1020 and pex-tool#1108 by equipping PEXEnvironment with the ability to resolve the appropriate transitive set of distributions from a root set of requirements instead of the current full set of transitive requirements stored post-resolve in PexInfo.

This removes our dependency on pkg_resources Environment / WorkingSet in favor of performing our own recursive resolve of runtime distributions to activate using distribution metadata. This fixes an old test bug noticed by Benjy but, more importanty, sets the stage to fix #899, #1020 and #1108 by equipping PEXEnvironment with the ability to resolve the appropriate transitive set of distributions from a root set of requirements instead of the current full set of transitive requirements stored post-resolve in PexInfo.

In order to fix pex-tool#899, we need to map local project requirements to built wheels after the fact in order to learn the project name and version and thus the full requirement needed for PexInfo metadata. In order to do this with confidence, break up the ReqInfo type into three seperate types to cover the three main parse cases.

Previously we recorded the post-resolve exact requirements of all resolved distributions. This necessitated a good deal of work to recover environment markers and still left out any top-level environment markers specified in direct requirements / requirement files passed to Pex. Now we record all requirements as we received them, only filling in project name and version for local loose-source requirements in our resolve post-processing. This saves time (~O(500ms) for medium to large PEX builds) and fixes bugs. Fixes pex-tool#899

In order to fix #899, we need to map local project requirements to built wheels after the fact in order to learn the project name and version and thus the full requirement needed for PexInfo metadata. In order to do this with confidence, break up the ReqInfo type into three seperate types to cover the three main parse cases.

Previously we recorded the post-resolve exact requirements of all resolved distributions. This necessitated a good deal of work to recover environment markers and still left out any top-level environment markers specified in direct requirements / requirement files passed to Pex. Now we record all requirements as we received them, only filling in project name and version for local loose-source requirements in our resolve post-processing. This saves time (~O(500ms) for medium to large PEX builds) and fixes bugs. Fixes pex-tool#899

Previously we recorded the post-resolve exact requirements of all resolved distributions. This necessitated a good deal of work to recover environment markers and still left out any top-level environment markers specified in direct requirements / requirement files passed to Pex. Now we record all requirements as we received them, only filling in project name and version for local loose-source requirements in our resolve post-processing. This saves time (~O(500ms) for medium to large PEX builds) and fixes bugs. Fixes #899

jsirois added bug resolver labels Feb 20, 2020

jsirois mentioned this issue Feb 20, 2020

Check expected pex invocation failure reason in tests. #874

Merged

jsirois mentioned this issue Feb 20, 2020

Release 2.1.4 #893

Closed

1 task

jsirois mentioned this issue Feb 20, 2020

Release 2.1.5 #902

Closed

2 tasks

jsirois mentioned this issue Feb 27, 2020

Release 2.1.7 #909

Closed

6 tasks

jsirois mentioned this issue Mar 25, 2020

Release 2.1.8 #933

Closed

4 tasks

jsirois mentioned this issue Apr 2, 2020

Release 2.1.9 #944

Closed

1 task

benjyw mentioned this issue Apr 7, 2020

Release 2.1.10 #950

Closed

10 tasks

jsirois mentioned this issue May 11, 2020

Release 2.1.13 #972

Closed

3 tasks

jsirois mentioned this issue Jul 10, 2020

Release 2.1.16 #999

Closed

3 tasks

jsirois mentioned this issue Aug 29, 2020

Release 2.1.17 #1013

Closed

4 tasks

This was referenced Oct 2, 2020

Release 2.1.18 #1053

Closed

Release 2.1.19 #1069

Closed

Eric-Arellano mentioned this issue Oct 15, 2020

Release 2.1.21 #1078

Closed

7 tasks

jsirois mentioned this issue Nov 10, 2020

Release 2.1.22 #1111

Closed

4 tasks

jsirois mentioned this issue Nov 17, 2020

Implement full featured requirment parsing. #1114

Merged

This was referenced Dec 14, 2020

Release 2.1.24 #1138

Closed

Release 2.1.25 #1144

Closed

jsirois self-assigned this Jan 3, 2021

jsirois added the in progress label Jan 3, 2021

jsirois mentioned this issue Jan 4, 2021

Support project_name_and_version metadata. #1160

Merged

jsirois added a commit to jsirois/pex that referenced this issue Jan 4, 2021

Add support for parsing extras and specifiers.

1dd278c

Work towards pex-tool#899, pex-tool#1020 and pex-tool#1108.

jsirois mentioned this issue Jan 4, 2021

Add support for parsing extras and specifiers. #1161

Merged

jsirois added a commit that referenced this issue Jan 5, 2021

Add support for parsing extras and specifiers. (#1161)

de5f491

Work towards #899, #1020 and #1108.

jsirois mentioned this issue Jan 8, 2021

PEXEnvironment recursive runtime resolve. #1165

Merged

jsirois closed this as completed in #1165 Jan 8, 2021

jsirois reopened this Jan 9, 2021

This was referenced Jan 12, 2021

Type SpawnedJob. #1167

Merged

Type AtomicDirectory. #1168

Merged

Type BuildAndInstallRequest. #1169

Merged

jsirois mentioned this issue Jan 12, 2021

Tighten requirements parsing. #1170

Merged

jsirois closed this as completed in #1170 Jan 13, 2021

jsirois mentioned this issue Jan 13, 2021

Record original requirements in PEX-INFO. #1171

Merged

jsirois removed the in progress label Jan 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Top level environment markers are lost in built PEXes. #899

Top level environment markers are lost in built PEXes. #899

jsirois commented Feb 20, 2020

jsirois commented Feb 20, 2020

jsirois commented Feb 20, 2020 •

edited

Loading

jhermann commented Feb 20, 2020

jsirois commented Feb 26, 2020

jhermann commented Feb 26, 2020

jsirois commented Mar 23, 2020

Top level environment markers are lost in built PEXes. #899

Top level environment markers are lost in built PEXes. #899

Comments

jsirois commented Feb 20, 2020

jsirois commented Feb 20, 2020

jsirois commented Feb 20, 2020 • edited Loading

jhermann commented Feb 20, 2020

jsirois commented Feb 26, 2020

jhermann commented Feb 26, 2020

jsirois commented Mar 23, 2020

jsirois commented Feb 20, 2020 •

edited

Loading