Add --scie option to produce native PEX exes.
#2466
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,202 @@ | ||
| # Copyright 2024 Pex project contributors. | ||
| # Licensed under the Apache License, Version 2.0 (see LICENSE). | ||
|
|
||
| from __future__ import absolute_import | ||
|
|
||
| import os.path | ||
| from argparse import Namespace, _ActionsContainer | ||
|
|
||
| from pex.compatibility import urlparse | ||
| from pex.fetcher import URLFetcher | ||
| from pex.orderedset import OrderedSet | ||
| from pex.pep_440 import Version | ||
| from pex.scie import science | ||
| from pex.scie.model import ( | ||
| ScieConfiguration, | ||
| ScieInfo, | ||
| ScieOptions, | ||
| SciePlatform, | ||
| ScieStyle, | ||
| ScieTarget, | ||
| ) | ||
| from pex.scie.science import SCIENCE_RELEASES_URL, SCIENCE_REQUIREMENT | ||
| from pex.typing import TYPE_CHECKING, cast | ||
| from pex.variables import ENV, Variables | ||
|
|
||
| if TYPE_CHECKING: | ||
| from typing import Iterator, Optional, Tuple, Union | ||
|
|
||
|
|
||
| __all__ = ( | ||
| "ScieConfiguration", | ||
| "ScieInfo", | ||
| "SciePlatform", | ||
| "ScieStyle", | ||
| "ScieTarget", | ||
| "build", | ||
| ) | ||
|
|
||
|
|
||
| def register_options(parser): | ||
| # type: (_ActionsContainer) -> None | ||
|
|
||
| parser.add_argument( | ||
| "--scie", | ||
| "--par", | ||
| dest="scie_style", | ||
| default=None, | ||
| type=ScieStyle.for_value, | ||
| choices=ScieStyle.values(), | ||
| help=( | ||
| "Create one or more native executable scies from your PEX that include a portable " | ||
| "CPython interpreter along with your PEX making for a truly hermetic PEX that can run " | ||
| "on machines with no Python installed at all. If your PEX has multiple targets, " | ||
| "whether `--platform`s, `--complete-platform`s or local interpreters in any " | ||
| "combination, then one PEX scie will be made for each platform, selecting the latest " | ||
| "compatible portable CPython interpreter. Note that only CPython>=3.8 is supported. If " | ||
| "you'd like to explicitly control the target platforms or the exact portable CPython " | ||
| "selected, see `--scie-platform`, `--scie-pbs-release` and `--scie-python-version`. " | ||
| "Specifying `--scie {lazy}` will fetch the portable CPython interpreter just in time " | ||
| "on first boot of the PEX scie on a given machine if needed. The URL(s) to fetch the " | ||
| "portable CPython interpreter from can be customized by exporting the " | ||
| "PEX_BOOTSTRAP_URLS environment variable pointing to a json file with the format: " | ||
| '`{{"ptex": {{<file name 1>: <url>, ...}}}}` where the file names should match those ' | ||
| "found via `SCIE=inspect <the PEX scie> | jq .ptex` with appropriate replacement URLs. " | ||
| "Specifying `--scie {eager}` will embed the portable CPython interpreter in your PEX " | ||
| "scie making for a larger file, but requiring no internet access to boot. If you have " | ||
| "customization needs not addressed by the Pex `--scie*` options, consider using " | ||
| "`science` to build your scies (which is what Pex uses behind the scenes); see: " | ||
| "https://science.scie.app.".format(lazy=ScieStyle.LAZY, eager=ScieStyle.EAGER) | ||
|
Comment on lines
+69
to
+72
There was a problem hiding this comment. @sureshjoshi I see that your Pants plugin accepts an optional custom lift manifest, parses it if present, then injects bits into it. I think to support that sort of thing in a principled way, I'd have to parse the user supplied manifest and confirm they do not set the following keys:
Additionally, I'd have to advertise that I bind ptex to "ptex" for lazy scies, and always bind Without all this I don't see how the user supplied manifest can work with Pex needs fruitfully. Can you think of any other corners? Perhaps I'm overthinking. Do you need this functionality? There was a problem hiding this comment. I guess for There was a problem hiding this comment. I was hacking around tonight, trying to envision how I'd re-build something like One idea was to manipulate the embedded manifest after In the case of the plugin (which, I wouldn't really use as a reference for anything - as I made it a few years ago to solve an immediate deployment problem on a client project), I think we try to use the optional lift.toml where possible and inject the target names under certain conditions. For this PR, I don't see any problems with deferring all of those concerns, but I'm of two minds.
Would it make sense/be possible for [lift]
name = "pantsible"
description = "Ansible with an embedded Python interpreter."
platforms = ... inferred from pex ...
[[lift.interpreters]] -> ... inferred from pex ...
[[lift.files]]
name = "pex"
[[lift.commands]]
name = "ansible"
exe = "{scie.bindings.venv}/venv/bin/ansible"
args = []
...Although, one immediate problem I see here... I think I'm conflating a There was a problem hiding this comment. Reading through the PR, another thought that popped into my head is allowing for the Whether that functionality is in There was a problem hiding this comment.
Well,
Well, science is general purpose - Any language; so it doesn't really make sense for it to know about Python let alone Pex. It does have a Provider interface to supply interpreters and that has exactly 1 implementation currently, that provides PBS interpreters. A PEX provider might make sense. That said, Pex creates PEXes - single file executables. These do not have:
As such, I think it makes sense for Pex to offer the ability to take your PEX file and turn it into a scie that behaves exactly the same, with nothing extra except maybe running faster. Everything you'd do in a custom manifest, afaict, would add things the PEX cannot already do. At that point, having to move up a layer and use science yourself with a custom lift manifest to build your app not using Pex directly makes sense. I.E.: what scie-pants has to do. The Pants app is more complex than just what the Pants PEX does / has tight perf overhead concerns; so it makes sense to move up to the higher layer.
That's exactly what I meant by all this: #2466 (comment) It seems to me you can't just overlay, you must confirm the key mechanisms Pex uses in its lift are not destroyed by the merge before merging. There was a problem hiding this comment. I'm referring to downstream tools like Anyways, the things I have in my mind are probably out of scope of this PR, and if they're important enough, or strongly enough use-cased, I can open a new ticket later. There was a problem hiding this comment. Gotcha. So I think the PEX interpreter Provider would just use the I.E.: not create the scie, but use the That said, the current science Provider interface only allows providing an interpreter and not a set of platforms; so new API work would need to be done in science anyhow it seems to plug all this in. There was a problem hiding this comment. I guess the current API does allow enough for a PEX interpreter Provider to error when asked to produce an interpreter distribution via [lift]
name = "example"
platforms = [
"linux-aarch64",
"linux-x86_64",
"macos-aarch64",
]
[[lift.files]]
name = "pex"
[[lift.interpreters]]
id = "cpython"
provider = "PEX"
pex = "{pex}"Here if I ran There was a problem hiding this comment. Yep, there we go - that's the kinda thing I see value in. One less place where head scratching can take place. There was a problem hiding this comment. The restricting use case for |
||
| ), | ||
| ) | ||
| parser.add_argument( | ||
| "--scie-platform", | ||
| dest="scie_platforms", | ||
| default=[], | ||
| action="append", | ||
| type=SciePlatform.for_value, | ||
| choices=SciePlatform.values(), | ||
| help=( | ||
| "The platform to produce the native PEX scie executable for. Can be specified multiple " | ||
| "times." | ||
| ), | ||
| ) | ||
| parser.add_argument( | ||
| "--scie-pbs-release", | ||
| dest="scie_pbs_release", | ||
| default=None, | ||
| type=str, | ||
| help=( | ||
| "The Python Standalone Builds release to use. Currently releases are dates of the form " | ||
| "YYYYMMDD, e.g.: '20240713'. See their GitHub releases page at " | ||
| "https://github.com/indygreg/python-build-standalone/releases to discover available " | ||
| "releases. If left unspecified the latest release is used. N.B.: The latest lookup is " | ||
| "cached for 5 days. To force a fresh lookup you can remove the cache at " | ||
| "<USER CACHE DIR>/science/downloads." | ||
| ), | ||
| ) | ||
| parser.add_argument( | ||
| "--scie-python-version", | ||
| dest="scie_python_version", | ||
| default=None, | ||
| type=Version, | ||
| help=( | ||
| "The portable CPython version to select. Can be either in `<major>.<minor>` form; " | ||
| "e.g.: '3.11', or else fully specified as `<major>.<minor>.<patch>`; e.g.: '3.11.3'. " | ||
| "If you don't specify this option, Pex will do its best to guess appropriate portable " | ||
| "CPython versions. N.B.: Python Standalone Builds does not provide all patch versions; " | ||
| "so you should check their releases at " | ||
| "https://github.com/indygreg/python-build-standalone/releases if you wish to pin down " | ||
| "to the patch level." | ||
| ), | ||
| ) | ||
| parser.add_argument( | ||
| "--scie-science-binary", | ||
| dest="scie_science_binary", | ||
| default=None, | ||
| type=str, | ||
| help=( | ||
| "The file path of a `science` binary or a URL to use to fetch the `science` binary " | ||
| "when there is no `science` on the PATH with a version matching {science_requirement}. " | ||
| "Pex uses the official `science` releases at {science_releases_url} by default.".format( | ||
| science_requirement=SCIENCE_REQUIREMENT, science_releases_url=SCIENCE_RELEASES_URL | ||
| ) | ||
| ), | ||
| ) | ||
|
|
||
|
|
||
| def render_options(options): | ||
| # type: (ScieOptions) -> str | ||
|
|
||
| args = ["--scie", str(options.style)] | ||
| for platform in options.platforms: | ||
| args.append("--scie-platform") | ||
| args.append(str(platform)) | ||
| if options.pbs_release: | ||
| args.append("--scie-pbs-release") | ||
| args.append(options.pbs_release) | ||
| if options.python_version: | ||
| args.append("--scie-python-version") | ||
| args.append(".".join(map(str, options.python_version))) | ||
| if options.science_binary_url: | ||
| args.append("--scie-science-binary") | ||
| args.append(options.science_binary_url) | ||
| return " ".join(args) | ||
|
|
||
|
|
||
| def extract_options(options): | ||
| # type: (Namespace) -> Optional[ScieOptions] | ||
|
|
||
| if not options.scie_style: | ||
| return None | ||
|
|
||
| python_version = None # type: Optional[Union[Tuple[int, int], Tuple[int, int, int]]] | ||
| if options.scie_python_version: | ||
| if ( | ||
| not options.scie_python_version.parsed_version.release | ||
| or len(options.scie_python_version.parsed_version.release) < 2 | ||
| ): | ||
| raise ValueError( | ||
| "Invalid Python version: '{python_version}'.\n" | ||
| "Must be in the form `<major>.<minor>` or `<major>.<minor>.<patch>`".format( | ||
| python_version=options.scie_python_version | ||
| ) | ||
| ) | ||
| python_version = cast( | ||
| "Union[Tuple[int, int], Tuple[int, int, int]]", | ||
| options.scie_python_version.parsed_version.release, | ||
| ) | ||
| if python_version < (3, 8): | ||
| raise ValueError( | ||
| "Invalid Python version: '{python_version}'.\n" | ||
| "Scies are built using Python Standalone Builds which only supports Python >=3.8.\n" | ||
| "To find supported Python versions, you can browse the releases here:\n" | ||
| " https://github.com/indygreg/python-build-standalone/releases".format( | ||
| python_version=options.scie_python_version | ||
| ) | ||
| ) | ||
|
|
||
| science_binary_url = options.scie_science_binary | ||
| if science_binary_url: | ||
| url_info = urlparse.urlparse(options.scie_science_binary) | ||
| if not url_info.scheme and url_info.path and os.path.isfile(url_info.path): | ||
| science_binary_url = "file://{path}".format(path=os.path.abspath(url_info.path)) | ||
|
|
||
| return ScieOptions( | ||
| style=options.scie_style, | ||
| platforms=tuple(OrderedSet(options.scie_platforms)), | ||
| pbs_release=options.scie_pbs_release, | ||
| python_version=python_version, | ||
| science_binary_url=science_binary_url, | ||
| ) | ||
|
|
||
|
|
||
| def build( | ||
| configuration, # type: ScieConfiguration | ||
| pex_file, # type: str | ||
| url_fetcher=None, # type: Optional[URLFetcher] | ||
| env=ENV, # type: Variables | ||
| ): | ||
| # type: (...) -> Iterator[ScieInfo] | ||
|
|
||
| return science.build(configuration, pex_file, url_fetcher=url_fetcher, env=env) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # Copyright 2024 Pex project contributors. | ||
| # Licensed under the Apache License, Version 2.0 (see LICENSE). | ||
|
|
||
| from __future__ import print_function | ||
|
|
||
| import os | ||
| import sys | ||
|
|
||
|
|
||
| def write_bindings( | ||
| env_file, # type: str | ||
| installed_pex_dir, # type: str | ||
| ): | ||
| # type: (...) -> None | ||
| with open(env_file, "a") as fp: | ||
| print("PYTHON=" + sys.executable, file=fp) | ||
| print("PEX=" + os.path.realpath(os.path.join(installed_pex_dir, "__main__.py")), file=fp) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| write_bindings( | ||
| env_file=os.environ["SCIE_BINDING_ENV"], | ||
| installed_pex_dir=( | ||
| # The zipapp case: | ||
| os.environ["_PEX_SCIE_INSTALLED_PEX_DIR"] | ||
jsirois marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # The --venv case: | ||
| or os.environ.get("VIRTUAL_ENV", os.path.dirname(os.path.dirname(sys.executable))) | ||
| ), | ||
| ) | ||
| sys.exit(0) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As the issues attached to this PR prove, people in the world know "PAR"; so it seems to make sense to add a
--paralias to this one option for discoverability by those people. If they need more than the default--partreatment, then they really must learn about scies and--scie-*advanced options anyhow.