Enhance runtime error messages for platform-specific incompatibilities

[CMLivingston]

This is a pex-specific ticket for the feature proposal in: pantsbuild/pants#8031

To sum up, the current pex resolver error thrown at runtime does not provide users with enough information as to what phase of the pex execution process creates a given runtime failure a la:

Failed to execute PEX file, missing macosx_10_13_x86_64-cp-36-cp36m compatible dependencies for:
cffi
cryptography
pyyaml

The key points for improvement are summarized in the linked ticket:

There seem to be a few bits that are critical: If PEX is in --build mode (it is by default) and it builds any platform specific wheels, it needs to note this and write down relevant constraints in its metadata noting the resulting pex for the same root platform can only run under said constraint. If PEX resolves a platform-specific wheel (as opposed to building it as in 1 above), it must use the resolved wheel platform info to update similar constraints to above.

The proposal here is to ensure that we provide error messaging and appropriate context based on what step of the bootstrapping process introduced the runtime interpreter/dependency incompatibility.

[jsirois]

@CMLivingston since you have a user base that feels this pain point acutely, perhaps you can poll them and suggest some exact wording / layout for the error message on the runtime "resolve" case you outline here. That would provide a nice concrete point for implementation to launch from as well as provide space for any required debate on messaging here in the issue.

[jsirois]

I think this can be closed as implemented in the absence of further details about the original request. Both the PEX bootstrap behavior (trying all available interpreters for bootstrapping if need be before giving up) and the error message seem quite a bit more clear now.

Pex build time failure in "yolo" mode ¹

We now attempt to build sdists when no platform-specific wheel is compatible on the chance the wheel is not platform-specific. When that doesn't work you now see:

:; pex --complete-platform package/complete-platforms/linux-aarch64.json p537
No pre-built wheel was available for p537 1.0.7.
Successfully built the wheel p537-1.0.7-cp311-cp311-linux_x86_64.whl from the sdist p537-1.0.7.tar.gz but it is not compatible with the requested foreign target complete platform cp312-cp312-manylinux_2_31_aarch64.
You'll need to build a wheel from p537-1.0.7.tar.gz on the foreign target platform and make it available to Pex via a `--find-links` repo or a custom `--index`.

Build OK, deploy environment mismatch ²

As an example, if I build the Pex PEX targeting macOS arm64:

:; pex --complete-platform package/complete-platforms/macos-aarch64.json --lock package/pex-scie.lock -opex.pex

And then I try to run it on linux x86-64, I now get:

:; ./pex.pex
Failed to find compatible interpreter on path /home/jsirois/.pyenv/libexec:/home/jsirois/.pyenv/plugins/python-build/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.local/bin:/home/jsirois/bin:/home/jsirois/.pyenv/bin:/home/jsirois/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/snap/bin.

Examined the following working interpreters:
1.)                                         /usr/bin/python3.12 CPython==3.12.3
2.)       /home/jsirois/.pyenv/versions/pypy2.7-7.3.17/bin/pypy PyPy==2.7.18
3.)  /home/jsirois/.pyenv/versions/pypy3.10-7.3.17/bin/pypy3.10 PyPy==3.10.14
4.)       /home/jsirois/.pyenv/versions/pypy3.6-7.3.3/bin/pypy3 PyPy==3.6.12
5.)       /home/jsirois/.pyenv/versions/pypy3.7-7.3.9/bin/pypy3 PyPy==3.7.13
6.)      /home/jsirois/.pyenv/versions/pypy3.8-7.3.11/bin/pypy3 PyPy==3.8.16
7.)    /home/jsirois/.pyenv/versions/pypy3.9-7.3.16/bin/pypy3.9 PyPy==3.9.19
8.)        /home/jsirois/.pyenv/versions/3.11.10/bin/python3.11 CPython==3.11.10
9.)          /home/jsirois/.pyenv/versions/2.7.18/bin/python2.7 CPython==2.7.18
10.) /home/jsirois/.pex_dev/pyenv/versions/3.10.7/bin/python3.10 CPython==3.10.7
11.)      /home/jsirois/.pyenv/versions/3.13.0rc2/bin/python3.13 CPython==3.13.0
12.)          /home/jsirois/.pyenv/versions/3.5.10/bin/python3.5 CPython==3.5.10
13.)          /home/jsirois/.pyenv/versions/3.6.15/bin/python3.6 CPython==3.6.15
14.)          /home/jsirois/.pyenv/versions/3.7.17/bin/python3.7 CPython==3.7.17
15.)          /home/jsirois/.pyenv/versions/3.8.20/bin/python3.8 CPython==3.8.20
16.)          /home/jsirois/.pyenv/versions/3.9.20/bin/python3.9 CPython==3.9.20

Skipped the following broken interpreters:
1.) /home/jsirois/.pyenv/versions/pypy3.5-7.0.0/bin/pypy3.5:
ExecutionError('OSError(8, \'Exec format error\') while trying to execute `[\'/home/jsirois/.pyenv/versions/pypy3.5-7.0.0/bin/pypy3.5\', \'-s\', \'-c\', "import os\\nimport sys\\n\\nfrom pex.atomic_directory import atomic_directory\\nfrom pex.common import safe_open\\nfrom pex.interpreter import PythonIdentity\\n\\n\\nencoded_identity = PythonIdentity.get(binary=\'/home/jsirois/.pyenv/versions/pypy3.5-7.0.0/bin/pypy3.5\').encode()\\nwith atomic_directory(\'/home/jsirois/.cache/pex/interpreters/0/2da2fa5e75b1fa229b1b26fa0aa14d143315f137/b0bea01d58a5bbdcc8b6b4c9961654212b99f90e/57743e656901d45a2c9ccb0a97892c8bdeb9daaf\') as cache_dir:\\n    if not cache_dir.is_finalized():\\n        with safe_open(\\n            os.path.join(cache_dir.work_dir, \'INTERP-INFO\'), \'w\'\\n        ) as fp:\\n            fp.write(encoded_identity)\\n"]`')

(See https://github.com/pex-tool/pex/issues/1027 for a list of known breaks and workarounds.)

No working interpreter compatible with the requested constraints was found:

  A distribution for psutil could not be resolved for /usr/bin/python3.12.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /usr/bin/python3.12:
  cp312-cp312-manylinux_2_39_x86_64
  ... 1067 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/pypy2.7-7.3.17/bin/pypy.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/pypy2.7-7.3.17/bin/pypy:
  pp27-pypy_73-manylinux_2_39_x86_64
  ... 437 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/pypy3.10-7.3.17/bin/pypy3.10.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/pypy3.10-7.3.17/bin/pypy3.10:
  pp310-pypy310_pp73-manylinux_2_39_x86_64
  ... 558 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/pypy3.6-7.3.3/bin/pypy3.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/pypy3.6-7.3.3/bin/pypy3:
  pp36-pypy36_pp73-manylinux_2_39_x86_64
  ... 398 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/pypy3.7-7.3.9/bin/pypy3.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/pypy3.7-7.3.9/bin/pypy3:
  pp37-pypy37_pp73-manylinux_2_39_x86_64
  ... 438 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/pypy3.8-7.3.11/bin/pypy3.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/pypy3.8-7.3.11/bin/pypy3:
  pp38-pypy38_pp73-manylinux_2_39_x86_64
  ... 478 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/pypy3.9-7.3.16/bin/pypy3.9.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/pypy3.9-7.3.16/bin/pypy3.9:
  pp39-pypy39_pp73-manylinux_2_39_x86_64
  ... 518 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/3.11.10/bin/python3.11.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/3.11.10/bin/python3.11:
  cp311-cp311-manylinux_2_39_x86_64
  ... 988 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/2.7.18/bin/python2.7.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/2.7.18/bin/python2.7:
  cp27-cp27mu-manylinux_2_39_x86_64
  ... 437 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pex_dev/pyenv/versions/3.10.7/bin/python3.10.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pex_dev/pyenv/versions/3.10.7/bin/python3.10:
  cp310-cp310-manylinux_2_39_x86_64
  ... 909 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/3.13.0rc2/bin/python3.13.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/3.13.0rc2/bin/python3.13:
  cp313-cp313-manylinux_2_39_x86_64
  ... 1146 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/3.5.10/bin/python3.5.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/3.5.10/bin/python3.5:
  cp35-cp35m-manylinux_2_39_x86_64
  ... 513 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/3.6.15/bin/python3.6.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/3.6.15/bin/python3.6:
  cp36-cp36m-manylinux_2_39_x86_64
  ... 592 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/3.7.17/bin/python3.7.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/3.7.17/bin/python3.7:
  cp37-cp37m-manylinux_2_39_x86_64
  ... 672 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/3.8.20/bin/python3.8.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/3.8.20/bin/python3.8:
  cp38-cp38-manylinux_2_39_x86_64
  ... 751 more ...

  A distribution for psutil could not be resolved for /home/jsirois/.pyenv/versions/3.9.20/bin/python3.9.
  Found 1 distribution for psutil that do not apply:
  1.) The wheel tags for psutil 6.0.0 are cp38-abi3-macosx_11_0_arm64 which do not match the supported tags of /home/jsirois/.pyenv/versions/3.9.20/bin/python3.9:
  cp39-cp39-manylinux_2_39_x86_64
  ... 830 more ...

Footnotes

1. Yolo mode was spec'd in Support attempts to build sdists for foreign platforms. #2073, implemented in Attempt "cross-builds" of sdists for foreign platforms. #2075 and released in https://github.com/pex-tool/pex/releases/tag/v2.1.125 ↩

2. Robust runtime interpreter selection was spec'd in Decouple PEX runtime interpreter selection from buildtime interpreter selection. #1020, implemented in Select PEX runtime interpreter robustly. #1770 and released in https://github.com/pex-tool/pex/releases/tag/v2.1.89 ↩