Implement long-term solution for cross-interpreter requirement handling the in pex resolver

[CMLivingston]

When a setup.py file specifies install_requires requirements with interpreter-specific constraints, the pex resolver cannot parse and handle these constraints to produce a cross-interpreter pex. See https://github.com/Julian/jsonschema/blob/master/setup.py#L43 for an example of such a constraint.

Interpreter constraints like this are typically for stdlib backport libraries and the like that only make sense to include in a universal dist, so this problem is not as significant when the pex is targeting a specific interpreter version. In the targeted case, the resolver only needs to ignore requirements that are pinned to non-targeted interpreter versions. The short term workaround for the targeted-interpreter case is to allow the caller of pex.resolver#resolve to supply a blacklist dict that maps package name string -> interpreter constraint string, allowing the resolver to effectively skip them as implemented in #457.

The scope of this ticket is to remove the blacklist implementation and replace it with smarter handling for the "pex-targeting-X-interpreter-version" case, specifically where all requirements are compared against interpreter constraints for a pex. Then at run time, the pex environment selectively activates packaged dependencies in accordance with a given pex's interpreter constraints. If no constraints are specified and the pex is intended to run with Python 2 or 3, then there should be support for determining this at runtime and handling env activation accordingly as well.

This approach would solve the cross-interpreter case. This is a similar problem to the one surfaced in #455, however that issue deals with cross-platform specifications, but this is effectively the same concept.

[kmannislands]

Is there any workaround for this currently available?
I tried something like:

$ pex jupyter --python=python2.7 --python=python3.4 -o jup_pex.pex

Which results in:

**** Failed to install functools32-3.2.3-2 (caused by: NonZeroExit("received exit code 1 during execution of `['/Users/kieran/.pyenv/versions/3.4.8/bin/python3.4', '-', 'bdist_wheel', '--dist-dir=/var/folders/7d/cnv90sjj7rx4nhq38nvx44nr0000gn/T/tmpZEG5fd']` while trying to execute `['/Users/kieran/.pyenv/versions/3.4.8/bin/python3.4', '-', 'bdist_wheel', '--dist-dir=/var/folders/7d/cnv90sjj7rx4nhq38nvx44nr0000gn/T/tmpZEG5fd']`",)
):
stdout:

stderr:
This backport is for Python 2.7 only.


Traceback (most recent call last):
  File "/Users/kieran/.pyenv/versions/2.7.13/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/bin/pex.py", line 663, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/bin/pex.py", line 601, in build_pex
    for dist in resolveds:
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/resolver.py", line 438, in resolve_multi
    allow_prereleases):
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/resolver.py", line 376, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder))
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/resolver.py", line 209, in resolve
    dist = self.build(package, resolvable.options)
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/resolver.py", line 282, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/resolver.py", line 177, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage('file:///Users/kieran/.pex/build/functools32-3.2.3-2.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

Due to, I believe, the specific line in jsonschema's setup py (a nested subdependency of jupyter) that you referenced above.

Tried flattering my dependencies and trying to specify a environment marker directly in requirements.txt in an attempt to help pex's resolver out, so something like:

$ echo "functools32;python_version<'3.0'\njupyter" >> requirements.txt
$ pex -r requirements.txt --python=python2.7 --python=python3.4 -o jup_pex.pex

Which lead to the same error. So it would seem that pex fails to parse environment markers in requirements.txt (PEP508) as well as in install_requires of sub-packages.

Anyone have a different work around idea?

[CMLivingston]

I was unable to reproduce the error with that command, but it appears to be the same issue that I am describing here. There is no workaround that I am aware of, so unfortunately you will need to either build multiple pex files or target only one interpreter for the time being.

[kmannislands]

Hm, wish I could.
My use case is bundling python dependencies for a spark apps on a cluster where (clients, don't ask me why) some of the data nodes have varying versions of python installed.

My plan was to create a multi-interpreter pex file, ship it, and then specify PYSPARK_PYTHON=/path/to/multi_interpreter.pex, which would then act as the node's py interpreter, picking up on the actual python installed on the node but with the correct libraries depended on by a given app. Specifying different paths for nodes on a version dependent basis can be a bit messy.

Simple tests of this technique with SciPy worked, so I figured if it could bundle all that c and fortran, other dependencies would be fine. Whoops 😅

I'm still very interested in pursuing this approach. What does the timeline look like on this work?

I'll give the source a closer read and see if I can be of assistance as well.

[CMLivingston]

I see. In terms of a timeline on my availability to knock this out, I would say it would be ~1 month to a merged fix. The long-term solution boils down to providing the pex resolver module with the intelligence to parse environmental markers correctly and propagate its decisions to include/exclude a requirement through to the PEX-INFO metadata. That second part is to ensure that there are no requirements that are registered but actually excluded due to an incompatible marker.

[jsirois]

As discovered in pantsbuild/pants#6613 there is a bit more work needed here to filter out top-level requirements passed to pex. The work in #582 only hit transitive requirements.