Prototype: Expose scripts to path #1097

[jneuff]

This PR is just intended as a basis for discussion. It provides a prototype for the feature I suggested in #1097.

For this I added two command line arguments:

• --expose-entry-points to set entry points (e.g. pip:main) to be exposed on PATH.
• --expose-scripts to set scripts (e.g. pip) to be exposed on PATH.

Also, to be able to call the PEX entry point (path to PEX file or path to unzipped PEX) I needed to record the PEX file's shebang in PEX-INFO.

If any exposed_entry_points or exposed_scripts are defined, they will be rendered as scripts under $PEX_ROOT/bin/<hash>/<name>. Where <hash> ensures the directory is unique with respect to PEX shebang, PEX entry point and exposed entry points/scripts.

For a regular PEX exposing the script pip such a file might look like:

#!/bin/sh

PEX_SCRIPT=pip exec /usr/bin/env python3.8 /home/user/scratch/pex-1097/my.pex $@

Or for an unzipped PEX exposing the entry point pip:main:

#!/bin/sh

PEX_MODULE=pip:main exec /usr/bin/env python3.8 /home/user/.pex/unzipped_pexes/d7e4fb19eea99444d4ce8b85e5cffb4aca950bca $@

It turned out, the main problem was how to call the PEX. Because when running an unzipped PEX, the path to the original PEX file is lost. That is why I had to record the shebang in PEX-INFO, to being able to call an unzipped PEX the same way as running the PEX file.

The exposed entry points work with this prototype:

~/scratch/pex-1097 » ./my.pex                                                                                                                                                                                           d0301508@kaxpi00000122
Python 3.8.6 (default, Sep 25 2020, 00:00:00) 
[GCC 10.2.1 20200826 (Red Hat 10.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import subprocess
>>> subprocess.check_call("pip:main")
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.

Usage:   
  my.pex <command> [options]
..

Whereas exposed scripts do not:

>>> subprocess.check_call("pip")
Python 3.8.6 (default, Sep 25 2020, 00:00:00) 
[GCC 10.2.1 20200826 (Red Hat 10.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)

Calling pip dropped me into another Python interpreter. Probably, I broke resolving scripts. I'll dig into it.

Obviously, a lot is missing here, tests for instance. As I said this is just a prototype to serve as a basis for discussion. So, I'd be glad to get some feedback on this.

[jsirois]

Thanks for taking a stab at this! IIUC you don't need the --exposed-entry-points functionality, just the --exposed-scripts. If so, I have a change coming this weekend to support #962 that creates a venv on from a PEX at runtime. To do this, all the .whl bin/ scripts that are python scripts (console scripts), are re-written to have a shebang that is the selected runtime interpreter. You should probably wait on tht to land and see if it doesn't solve your use case. I'll include you on the review.

[jsirois]

This is not ideal. For example, it would fail in any Python distroless image whereas the corresponding python console script it wraps would not.

[jsirois]

Pex still supports Python 2.7 so you need to stick to those APIs - this will fail the 2.7 CI shards as-is.

[jsirois]

This hashing scheme is foiled by setting an entry_point for the module 'pex' today:
https://github.com/pantsbuild/pex/blob/c7c74cc607b83635063abc68cd613fb848b895d3/pex/__main__.py#L1-L8
and then changing that to be the 'pex' console script tomorrow:
https://github.com/pantsbuild/pex/blob/c7c74cc607b83635063abc68cd613fb848b895d3/pyproject.toml#L33-L34
You might form a dict using the key names corresponding to the 4 slots and then hash its sorted key json encoding.

[jsirois]

@jneuff if you have time to look at #1128 and in particular this test working backwards to the --bin-path feature it tests, I'd be grateful: https://github.com/pantsbuild/pex/pull/1128/files#diff-f6529bfd4644104d30930f6bf2eeafdf75f87ae12659a9185ac45c3fcdb2b8f9R180

[jneuff]

@jsirois Thanks for the feedback. Your venv feature sounds quite interesting! That should probably cover my use-cases. I'll take a look at your PR.

[jneuff]

I am closing this PR as the intended use-cases are well covered by the new venv pex-tool feature.

[jsirois]

Thanks for taking a stab at this @jneuff - it was definitely part of the motivation to finally implement #962 and solve #1097 along with other common issues running applications as PEX zip files.