Add `--scie-only` & `--scie-name-style`. #2523

jsirois · 2024-09-10T05:31:43Z

These options allow the files output when using --scie {eager,lazy}
scies to be better controlled. The --scie-only option provides a way
to ensure no PEX file is output, just the resulting scie. The
--scie-name-style option allows control over how the scie's target
platform is incorporated in the scie file name.

These options allow the files output when using `--scie {eager,lazy}` scies to be better controlled. The `--scie-only` option provides a way to ensure no PEX file is output, just the resulting scie. The `--scie-name-style` option allows control over how the scie's target platform is incorporated in the scie file name.

These are likely better than the Pex PEX for most Pex binary use cases since they include the management extra and run in `--venv` mode as well as providing BusyBox support for equal footing of the `pex`, `pex-tools` and `pex3` top-level commands.

jsirois · 2024-09-10T07:16:16Z

@martina-oefelein I'm interested in your thoughts on the new --scie-only and --scie-name-style options. The bits you are likely most interested in ordered from most relevant to least:

New options: d570fb9#diff-bbf96d2c6fdcaa284ebb9e1fc92f6485b122a18e9ed241d96e943c5a90fbe168R86-R115
Tests of the new options: d570fb9#diff-9d49325ca5168fedd20faae90ae081421469df69d073fd83be97c9d7ebcaad3cR1085-R1125
Pex scie release dogfood of the new options: 225dda6#diff-312c2b9fe952517b20c422ea9493fe918be43dc68b8e47233c2d1990e9bdcbbeR109-R117

I think this incorporates all your feedback from #2516 given the hard requirement I have of not breaking backwards compatibility; i.e: I could not change how things work by default and, unfortunately, I could only add options to get the desired results.

jsirois · 2024-09-10T07:18:49Z

Pants reviewers: this is not as bad as the diffstat looks. The change in da66a85 was machine-generated and accounts for the lions share of the delta lines:

    Run `tox -e gen-scie-platform -- --all`.

 package/complete-platforms/linux-aarch64.json |  496 ++++++++++
 package/complete-platforms/linux-x86_64.json  |  874 +++++++++++++++++
 package/complete-platforms/macos-aarch64.json |  604 ++++++++++++
 package/complete-platforms/macos-x86_64.json  | 2629 ++++++++++++++++++++++++++++++++++++++++++++++++++
 package/pex-scie.lock                         |  138 +++
 5 files changed, 4741 insertions(+)

jsirois · 2024-09-10T07:23:16Z

scripts/create-packages.py

+            "--no-build",
+            "--no-compile",
+            "--no-use-system-time",
+            "--venv",


One reason to switch from Pex PEX to a Pex scie:

:; hyperfine -w2 'dist/pex -V' 'dist/pex-linux-x86_64 pex -V' Benchmark 1: dist/pex -V Time (mean ± σ): 472.5 ms ± 7.7 ms [User: 445.7 ms, System: 35.4 ms] Range (min … max): 460.9 ms … 482.2 ms 10 runs Benchmark 2: dist/pex-linux-x86_64 pex -V Time (mean ± σ): 171.1 ms ± 5.2 ms [User: 163.4 ms, System: 10.8 ms] Range (min … max): 163.5 ms … 180.7 ms 17 runs Summary dist/pex-linux-x86_64 pex -V ran 2.76 ± 0.09 times faster than dist/pex -V

I could never do this with the Pex PEX since some host Pythons don't support -m venv out of the box (for example - Debian Pythons without an extra python-venv package installed). With a scie though, I know the PBS 3.12.5 CPython I've picked has -m venv.

The other reason is to easily gain useful pex3 cache purge output when the operation has to wait for in-flight PEX runs, but this may not be of much use depending on how Pants currently handles real-time console output from processes it launches.

Add a `--scie-busybox-pex-entrypoint-env-passthrough` option to trade a small amount of perf for a Pex scie busybox accepting PEX_{INTERPRETER,MODULE,SCRIPT} entry point control like its Pex PEX sibling.

jsirois · 2024-09-10T08:20:58Z

scripts/create-packages.py

+            "--scie-busybox",
+            "@pex",
+            "--scie-busybox-pex-entrypoint-env-passthrough",


N.B.: This means the released Pex scies, unlike the Pex PEX, support all 3 top-level Pex commands 1st class if you leave the file name as-is. For example:

:; dist/pex-linux-x86_64 Error: Could not determine which command to run. Please select from the following boot commands: pex pex-tools pex3 You can select a boot command by setting the SCIE_BOOT environment variable or else by passing it as the 1st argument.

So you can:

# Run `pex`: :; dist/pex-linux-x86_64 pex ... # Run `pex3`: :; dist/pex-linux-x86_64 pex3 ... # Run `pex-tools`: :; dist/pex-linux-x86_64 pex-tools ...

Alternatively you can do something like:

# Install the binaries with top-level names (the `--symlink` is optional): :; SCIE=install dist/pex-linux-x86_64 --symlink bin # Run `pex`: :; bin/pex ... # Run `pex3`: :; bin/pex3 ... # Run `pex-tools`: :; bin/pex-tools ...

Finally, if you re-name the downloaded Pex scie to pex, it behaves ~just like the Pex PEX and you must use env vars to access the alternate entry points; e.g: PEX_SCRIPT=pex3 pex ....

jsirois · 2024-09-10T08:36:22Z

.github/workflows/release.yml

+      - name: Append Hashes to Changelog
+        run: |
+          changelog_tmp="$(mktemp)"
+          cat "${{ steps.prepare-changelog.outputs.changelog-file }}" <(echo '***') dist/hashes.md \
+            > "${changelog_tmp}"
+          mv "${changelog_tmp}" "${{ steps.prepare-changelog.outputs.changelog-file }}"


The upshot of this is the Releases will get this style content between the change-log and the files list below:

file sha256 size

pex b5fb8b2469dd8f208a7805c7ec51850e81f4a4952e4808827b8e07589bc2496f 4306002

pex-linux-aarch64 1861382d2c273105a4b7013054e7acbab02f9ce78de7baa7d5e744c398eb4204 24854091

pex-linux-x86_64 e42f3231f605147ccb93118c2bb1386ca95dfac624b19cb405e91db9886cd7ad 28404512

pex-macos-aarch64 b1f18587833c9af437449dddf53dc778b1705eaace750b0f02ecee3bba5d2103 22320777

pex-macos-x86_64 fb8e9c580a9dc7ed0aae9f5b4ecad44287d4db585980400c2c3893687e545e74 22816650

benjyw

Thanks for the explanations throughout, will be helpful when integrating the PEX scies.

benjyw · 2024-09-12T00:08:34Z

pex/bin/pex.py

@@ -1375,6 +1375,8 @@ def do_main(
                        ),
                        V=options.verbosity,
                    )
+                if scie_configuration.options.scie_only:
+                    os.unlink(pex_file)


I assume this means that in certain cases of early failure the .pex file will be left behind as cruft. I imagine that is fine, but @martina-oefelein may have thoughts.

Yes, exactly. The existing scie creation code also goes to lengths to preserve the temporary lift.toml it generates when there is a scie building failure; so this intent to leave behind the inputs to the failure for inspection is consistent. In the lift.toml case the fact that it is being plopped out is actually printed to the console.

@martina-oefelein I'll likely submit this tomorrow morning ~9am U.S. Pacific unless I hear negative feedback from you. One thing I know is missing is docs and I'll eventually circle back to that. My main thrust here was to enable your use case (and Pex's, which matches yours) and to get a release out that Pants can use to handle overly huge Pex cache sizes there with the new pex3 cache ... commands that are only really nice to use with the Pex scie.

Sorry, I was offline. Yes, I think leaving the .pex file behind on failure is fine.

…e/pex-scies

jsirois · 2024-09-12T19:25:20Z

Alrighty, now available here: https://github.com/pex-tool/pex/releases/tag/v2.18.0

jsirois force-pushed the release/pex-scies branch from 8b9ccce to 8221e1c Compare September 10, 2024 06:00

jsirois added 4 commits September 10, 2024 00:05

Run tox -e gen-scie-platform -- --all.

da66a85

Package Pex --scies for release.

225dda6

These are likely better than the Pex PEX for most Pex binary use cases since they include the management extra and run in `--venv` mode as well as providing BusyBox support for equal footing of the `pex`, `pex-tools` and `pex3` top-level commands.

Prep the 2.18.0 release.

70c0c9d

jsirois force-pushed the release/pex-scies branch from 8221e1c to 70c0c9d Compare September 10, 2024 07:05

jsirois requested review from kaos, benjyw and huonw September 10, 2024 07:16

jsirois commented Sep 10, 2024

View reviewed changes

Support PEX_SCRIPT=pex3 ./pex ... for scie.

1d2dbb2

Add a `--scie-busybox-pex-entrypoint-env-passthrough` option to trade a small amount of perf for a Pex scie busybox accepting PEX_{INTERPRETER,MODULE,SCRIPT} entry point control like its Pex PEX sibling.

jsirois commented Sep 10, 2024

View reviewed changes

Clarify --scie-busybox-pex-entrypoint-env-passthrough help.

bdc61bd

jsirois requested a review from sureshjoshi September 10, 2024 08:29

jsirois commented Sep 10, 2024

View reviewed changes

Upgrade the Pex scie to 3.12.6.

531f7cf

benjyw approved these changes Sep 12, 2024

View reviewed changes

jsirois added 2 commits September 12, 2024 07:50

Merge remote-tracking branch 'refs/remotes/upstream/main' into releas…

f763917

…e/pex-scies

Fix release & update change log.

ae3edc8

jsirois merged commit d8a60db into pex-tool:main Sep 12, 2024
26 checks passed

jsirois deleted the release/pex-scies branch September 12, 2024 18:00

jsirois mentioned this pull request Sep 12, 2024

Add support for --no-use-platform-suffix. a-scie/lift#96

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `--scie-only` & `--scie-name-style`. #2523

Add `--scie-only` & `--scie-name-style`. #2523

jsirois commented Sep 10, 2024

jsirois commented Sep 10, 2024 •

edited

Loading

jsirois commented Sep 10, 2024

jsirois Sep 10, 2024 •

edited

Loading

jsirois Sep 10, 2024 •

edited

Loading

jsirois Sep 10, 2024

benjyw left a comment

benjyw Sep 12, 2024

jsirois Sep 12, 2024

jsirois Sep 12, 2024

martina-oefelein Sep 18, 2024

jsirois commented Sep 12, 2024

file	sha256	size
pex	b5fb8b2469dd8f208a7805c7ec51850e81f4a4952e4808827b8e07589bc2496f	4306002
pex-linux-aarch64	1861382d2c273105a4b7013054e7acbab02f9ce78de7baa7d5e744c398eb4204	24854091
pex-linux-x86_64	e42f3231f605147ccb93118c2bb1386ca95dfac624b19cb405e91db9886cd7ad	28404512
pex-macos-aarch64	b1f18587833c9af437449dddf53dc778b1705eaace750b0f02ecee3bba5d2103	22320777
pex-macos-x86_64	fb8e9c580a9dc7ed0aae9f5b4ecad44287d4db585980400c2c3893687e545e74	22816650

Add --scie-only & --scie-name-style. #2523

Add --scie-only & --scie-name-style. #2523

Conversation

jsirois commented Sep 10, 2024

jsirois commented Sep 10, 2024 • edited Loading

jsirois commented Sep 10, 2024

jsirois Sep 10, 2024 • edited Loading

Choose a reason for hiding this comment

jsirois Sep 10, 2024 • edited Loading

Choose a reason for hiding this comment

jsirois Sep 10, 2024

Choose a reason for hiding this comment

benjyw left a comment

Choose a reason for hiding this comment

benjyw Sep 12, 2024

Choose a reason for hiding this comment

jsirois Sep 12, 2024

Choose a reason for hiding this comment

jsirois Sep 12, 2024

Choose a reason for hiding this comment

martina-oefelein Sep 18, 2024

Choose a reason for hiding this comment

jsirois commented Sep 12, 2024

Add `--scie-only` & `--scie-name-style`. #2523

Add `--scie-only` & `--scie-name-style`. #2523

jsirois commented Sep 10, 2024 •

edited

Loading

jsirois Sep 10, 2024 •

edited

Loading

jsirois Sep 10, 2024 •

edited

Loading