Scie builds, --output, and resulting artifacts #2516

martina-oefelein · 2024-09-04T15:26:29Z

martina-oefelein
Sep 4, 2024

I'm trying to build scie executables for multiple platforms for my cli project as follows:

rm -r build/pex 
pex --no-build --requirement requirements-pex.txt --entry-point ardia.cli.ardia:main --package ardia@src --output-file build/pex/macosx_10_12_x86_64/ardia --scie eager --platform macosx_10_12_x86_64-cp-3.11.9-cp311
pex --no-build --requirement requirements-pex.txt --entry-point ardia.cli.ardia:main --package ardia@src --output-file build/pex/macosx_11_0_x86_64/ardia --scie eager --platform macosx_11_0_x86_64-cp-3.11.9-cp311
pex --no-build --requirement requirements-pex.txt --entry-point ardia.cli.ardia:main --package ardia@src --output-file build/pex/linux_x86_64/ardia --scie eager --platform linux_x86_64-cp-3.11.9-cp311
pex --no-build --requirement requirements-pex.txt --entry-point ardia.cli.ardia:main --package ardia@src --output-file build/pex/linux_aarch64/ardia --scie eager --platform linux_aarch64-cp-3.11.9-cp311

For the macOS builds, I get the expected executables, and they are named ardia according to the --output-file option.
However, for the linux builds, I get two files:

One named according to the --output-file option, but apparently without interpreter (just look at the size).
One with the PLATFORM part of --platform appended, which appears to be the desired scie executable:

>ls -l build/pex/*
build/pex/linux_aarch64:
total 76096
-rwxr-xr-x  1 moefelei  staff   5727439 Sep  4 17:02 ardia*
-rwxr-xr-x  1 moefelei  staff  33230479 Sep  4 17:02 ardia-linux-aarch64*

build/pex/linux_x86_64:
total 85016
-rwxr-xr-x  1 moefelei  staff   5999988 Sep  4 17:02 ardia*
-rwxr-xr-x  1 moefelei  staff  37527155 Sep  4 17:02 ardia-linux-x86_64*

build/pex/macosx_10_12_x86_64:
total 50024
-rwxr-xr-x  1 moefelei  staff  25611232 Sep  4 17:02 ardia*

build/pex/macosx_11_0_x86_64:
total 50024
-rwxr-xr-x  1 moefelei  staff  25611232 Sep  4 17:02 ardia*

Is this the expected behavior?
If so, this is rather confusing.

Answered by jsirois

Sep 4, 2024

This is currently the expected behavior - although I agree its confusing.

As an aside, one thing I will point out is that in your example above, you create two scies for the same platform (Mac x86-64), just with different Python versions. Since a scie contains its own Python, generally the end user doesn't care what Python version is embedded nor that the executable has anything to do with Python at all more generally. Perhaps you have a use case though where the user does care in some way?

Below I detail the current behavior and provide examples. Perhaps you can suggest a less confusing; more useful behavior or maybe I should just a a good documentation page to docs.pex-tool.org and / or…

View full answer

martina-oefelein · 2024-09-04T15:33:32Z

martina-oefelein
Sep 4, 2024
Author

Forgot to mention I'm using pex version 2.16.2

0 replies

jsirois · 2024-09-04T16:47:18Z

jsirois
Sep 4, 2024
Maintainer

This is currently the expected behavior - although I agree its confusing.

As an aside, one thing I will point out is that in your example above, you create two scies for the same platform (Mac x86-64), just with different Python versions. Since a scie contains its own Python, generally the end user doesn't care what Python version is embedded nor that the executable has anything to do with Python at all more generally. Perhaps you have a use case though where the user does care in some way?

Below I detail the current behavior and provide examples. Perhaps you can suggest a less confusing; more useful behavior or maybe I should just a a good documentation page to docs.pex-tool.org and / or improve --help.

The current intended output types behavior:

pex ... builds a PEX
pex --scie ... builds a PEX and, additionally, builds one or more scies from that PEX (you get >1 scie if you used multiple --scie-platform, --platform, --complete-platform or --python on the command line).

The current intended scie output naming behavior:

The scie name is derived from the PEX name by removing the .pex extension, if any. If there is no .pex extension, the PEX file is over-written by the scie created from it and you are left with just the scie if and only if the scie matches the current platform.
If the scie does not match the current platform, a platform extension is added.
When building multiple different platform scies at once, all scies will have their platform extension added, even a scie matching the current platform.

Some examples:

Using this project setup to mirror yours in structure:

:; cat requirements-pex.txt
ansicolors
cowsay

:; cat src/ardia/cli/ardia.py
import sys

import colors
import cowsay


def main() -> None:
    message = " ".join(sys.argv[1:])
    cowsay.tux(colors.cyan(message))

Note

I'm running under Linux x86-64 in these examples; so that's the current platform for me and mac is a "foreign" platform for me.

Current platform build with .pex extension (Here current platform is a local --python):

:; pex \
  --python python3.11 \
  --no-build \
  --requirement requirements-pex.txt \
  --entry-point ardia.cli.ardia:main \
  --package ardia@src \
  --scie eager \
  -o build/pex/ardia.pex

:; ls -1sh build/pex/
total 32M
 31M ardia
736K ardia.pex

:; file build/pex/*
build/pex/ardia:     ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, BuildID[sha1]=f1f01ca2ad165fed27f8304d4b2fad02dcacdffe, stripped
build/pex/ardia.pex: Zip archive data, made by v2.0 UNIX, extract using at least v2.0, last modified, last modified Sun, Jan 01 1980 00:00:00, uncompressed size 0, method=deflate

Current platform build with no extension (Here current platform is a --platform that happens to match a local python):

:; pex \
  --platform linux_x86_64-cp-3.11.9-cp311 \
  --no-build \
  --requirement requirements-pex.txt \
  --entry-point ardia.cli.ardia:main \
  --package ardia@src \
  --scie eager \
  -o build/pex/ardia

:; ls -1sh build/pex/
total 31M
31M ardia

:; file build/pex/*
build/pex/ardia: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, BuildID[sha1]=f1f01ca2ad165fed27f8304d4b2fad02dcacdffe, stripped

Foreign platform build with .pex extension:

:; pex \
  --platform macosx_11_0_x86_64-cp-3.11.9-cp311 \
  --no-build \
  --requirement requirements-pex.txt \
  --entry-point ardia.cli.ardia:main \
  --package ardia@src \
  --scie eager \
  -o build/pex/ardia.pex

:; ls -1sh build/pex/
total 21M
 20M ardia-macos-x86_64
736K ardia.pex

:; file build/pex/*
build/pex/ardia-macos-x86_64: Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE|HAS_TLV_DESCRIPTORS>
build/pex/ardia.pex:          Zip archive data, made by v2.0 UNIX, extract using at least v2.0, last modified, last modified Sun, Jan 01 1980 00:00:00, uncompressed size 0, method=deflate

Foreign platform build with no extension:

:; pex \
  --platform macosx_11_0_x86_64-cp-3.11.9-cp311 \
  --no-build \
  --requirement requirements-pex.txt \
  --entry-point ardia.cli.ardia:main \
  --package ardia@src \
  --scie eager \
  -o build/pex/ardia

:; ls -1sh build/pex/
total 21M
736K ardia
 20M ardia-macos-x86_64

:; file build/pex/*
build/pex/ardia:              Zip archive data, made by v2.0 UNIX, extract using at least v2.0, last modified, last modified Sun, Jan 01 1980 00:00:00, uncompressed size 0, method=deflate
build/pex/ardia-macos-x86_64: Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE|HAS_TLV_DESCRIPTORS>

Multi-platform build with no extension (with a .pex extension you get the same thing, but the PEX is named .pex just like -o asked).

:; pex \
  --platform linux_x86_64-cp-3.11.9-cp311 \
  --platform linux_aarch64-cp-3.11.9-cp311 \
  --platform macosx_11_0_x86_64-cp-3.11.9-cp311 \
  --no-build \
  --requirement requirements-pex.txt \
  --entry-point ardia.cli.ardia:main \
  --package ardia@src \
  --scie eager \
  -o build/pex/ardia

:; ls -1sh build/pex/
total 79M
736K ardia
 27M ardia-linux-aarch64
 31M ardia-linux-x86_64
 20M ardia-macos-x86_64

:; file build/pex/*
build/pex/ardia:               Zip archive data, made by v2.0 UNIX, extract using at least v2.0, last modified, last modified Sun, Jan 01 1980 00:00:00, uncompressed size 0, method=deflate
build/pex/ardia-linux-aarch64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=d55dd5a87a894376f51d578f879495586c907b8c, stripped
build/pex/ardia-linux-x86_64:  ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, BuildID[sha1]=f1f01ca2ad165fed27f8304d4b2fad02dcacdffe, stripped
build/pex/ardia-macos-x86_64:  Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE|HAS_TLV_DESCRIPTORS>

6 replies

jsirois Sep 4, 2024
Maintainer

I understand that you need to append platform suffixes to make the file names unique, but particularly for CLI tools, users would eventually want to rename them back, so that they can call it without suffix (and use the same command name on all platforms).
Maybe you could use separate directories instead of suffixes, like I tried in my example?

That makes sense of course. The model here though had been GitHub releases, where the structure is forced flat, like here: https://github.com/a-scie/lift/releases/tag/v0.7.0
An option to use platform dirs instead might make sense.

It is not very clear why there are both --platform and --scie-platform, and which one should use when.

In my first attempt, I tried --scie-platform, but it seems this affects only the interpreter executable. If you have compiled dependencies, you need to use --platform, which covers both interpreter and dependencies

Yeah, agreed. --scie-platform is only intended to narrow the list of platforms that scies are generated for.
For example, if you have:

pex \
  --platform macosx_10_12_x86_64-cp-3.11.9-cp311 \
  --platform linux_x86_64-cp-3.11.9-cp311 \
  --platform linux_aarch64-cp-3.11.9-cp311 \
  --scie-platform macos-x86_64 \
  -o app.pex

You'll build a multi-platform app.pex PEX zipapp that can run on all 3 platforms but only 1 scie - a macos-x86_64 scie. This is again on the idea that pex ... 1st and foremost always builds a PEX file. When you add --scie* that just adds a scie and changes behavior of that scie - it has no effect on the PEX that is built that actually powers the scie.

The syntax for --platform is rather complex. I assume it needs all this additional information to select the proper wheel, but the syntax isn't even the same (in particular order of fields is different)

It took a lot of trial and error to find out the correct syntax for platform. I think a few examples for the most common platforms would be very helpful.

The --help text for --platform links to PEP0427 and PEP0425, but these PEPs say that they're outdated. Maybe update to https://packaging.python.org/en/latest/specifications/binary-distribution-format/ instead?

Yeah, platform is an odd case. It grew at Twitter prior to standards and Pex won't break backwards compatibility so it remains. The best advice in that --help though is not to guess! It suggests using pex3 interpreter inspect and you should really do that. For example, to test a local Python 3.12 interpreter:

:; pex3 interpreter inspect --python python3.12 --verbose -i2
{
  "path": "/home/jsirois/dev/a-scie/lift/.venv/bin/python3.12",
  "version": "3.12.3",
  "requirement": "CPython==3.12.3",
  "sys_path": [
    "/usr/lib/python312.zip",
    "/usr/lib/python3.12",
    "/usr/lib/python3.12/lib-dynload",
    "/home/jsirois/dev/a-scie/lift/.venv/lib/python3.12/site-packages"
  ],
  "platform": "manylinux_2_39_x86_64-cp-3.12.3-cp312",
  "venv": true,
  "base_interpreter": "/usr/bin/python3.12"
}

So the platform for that one is manylinux_2_39_x86_64-cp-3.12.3-cp312.
There are examples in either pex --platform help or at https://docs.pex-tool.org/buildingpex.html#platform but agreed the documentation could be better.

That said, all those existing sources of doc try to point you to not using --platform and using --complete-platform instead. Any time you use --platform and it works, you got a bit lucky!

martina-oefelein Sep 5, 2024
Author

That said, all those existing sources of doc try to point you to not using --platform and using --complete-platform instead.

That didn't work for me at all. I got the impression that --complete-platform is an advanced thing.

If you like, I can try to help improving the documentation. May take a while, though.

jsirois Sep 9, 2024
Maintainer

It's more like --platform was an old mistake I can't get rid of (so as to not break existing users) and --complete-platform is the only thing that makes any sense at all. That said, it is slightly harder to create one, since you need one time access to a machine (or docker image) representative of the foreign platform so you can run pex3 interpreter inspect --markers --tags to collect the foreign complete platform. I'd be interested in the details of the error you encountered.

Towards that end, I'm currently working on releasing Pex scies and hit the same buggy naming behavior. I'll be posting a fix PR and adding you to the review for your thoughts. The idea is 2 new options:

--scie-only, --no-scie-only, --pex-and-scie
                        Only output a scie. By default, both a PEX and a scie
                        are output unless the `-o` / `--output-file` specified
                        has no '.pex' extension or a platform suffix is
                        included (see `--scie-name-platform`). (default:
                        False)
--scie-name-platform {dynamic,parent-dir,file-suffix}
                        Control how the `-o` / --output-file` translates to a 
                        scie name. By default, the platform is used as a file
                        suffix only when needed for disambiguation when targeting
                        a local platform. Specifying `--scie-name-platform file-suffix`
                        forces the scie target platform name to be added as a suffix of
                        the output filename; e.g.: `-o app` would produce a scie named
                        app-linux-x86_64 assuming the scie targets that platform.
                        Specifying `--scie-name-platform parent-dir` places the scie in
                        a sub-directory with the name of the platform it targets; e.g.:
                        `-o app` would produce a scie at `macos-aarch64/app` assuming
                        the scie targets that platform. (default: dynamic)

jsirois Sep 10, 2024
Maintainer

Alright, this is out for review in #2523.
Thanks for pushing this along @martina-oefelein.

jsirois Sep 13, 2024
Maintainer

Alrighty. With the 2.18.1 release your original post now works by using ... --output-file build/pex/ardia --scie-only --scie-name-style platform-parent-dir .... You get:

:; tree build
build
└── pex
    ├── linux-aarch64
    │   └── ardia
    ├── linux-x86_64
    │   └── ardia
    ├── macos-aarch64
    │   └── ardia
    └── macos-x86_64
        └── ardia

6 directories, 4 files

This is also immortalized in this integration test:

pex/tests/integration/scie/test_discussion_2516.py

Lines 24 to 98 in 1c9ac9e

    
           def test_discussion_2516_op(tmpdir): 
        
               # type: (Any) -> None 
        
               requirements = os.path.join(str(tmpdir), "requirements-pex.txt") 
        
               with open(requirements, "w") as fp: 
        
                   fp.write( 
        
                       dedent( 
        
                           """\ 
        
                           ansicolors 
        
                           cowsay 
        
                           """ 
        
                       ) 
        
                   ) 
        
               with safe_open(os.path.join(str(tmpdir), "src", "ardia", "cli", "ardia.py"), "w") as fp: 
        
                   fp.write( 
        
                       dedent( 
        
                           """\ 
        
                           import sys 
        
                           import colors 
        
                           import cowsay 
        
                           def main() -> None: 
        
                               message = " ".join(sys.argv[1:]) 
        
                               cowsay.tux(colors.cyan(message)) 
        
                           """ 
        
                       ) 
        
                   ) 
        
               out_dir = os.path.join(str(tmpdir), "build", "pex") 
        
               for abbreviated_platform in ( 
        
                   "linux_aarch64-cp-3.11.9-cp311", 
        
                   "linux_x86_64-cp-3.11.9-cp311", 
        
                   "macosx_11_0_arm64-cp-3.11.9-cp311", 
        
                   "macosx_11_0_x86_64-cp-3.11.9-cp311", 
        
               ): 
        
                   run_pex_command( 
        
                       args=[ 
        
                           "--no-build", 
        
                           "--requirement", 
        
                           "requirements-pex.txt", 
        
                           "--entry-point", 
        
                           "ardia.cli.ardia:main", 
        
                           "--package", 
        
                           "ardia@src", 
        
                           "--output-file", 
        
                           os.path.join(out_dir, "ardia"), 
        
                           "--scie", 
        
                           "eager", 
        
                           "--scie-only", 
        
                           "--scie-name-style", 
        
                           "platform-parent-dir", 
        
                           "--platform", 
        
                           abbreviated_platform, 
        
                       ], 
        
                       cwd=str(tmpdir), 
        
                   ).assert_success() 
        
               assert sorted( 
        
                   [ 
        
                       "build/pex/linux-aarch64/ardia", 
        
                       "build/pex/linux-x86_64/ardia", 
        
                       "build/pex/macos-aarch64/ardia", 
        
                       "build/pex/macos-x86_64/ardia", 
        
                   ] 
        
               ) == sorted( 
        
                   os.path.relpath(os.path.join(root, f), str(tmpdir)) 
        
                   for root, _, files in os.walk(out_dir) 
        
                   for f in files 
        
               ) 
        
               assert not glob.glob( 
        
                   os.path.join(str(tmpdir), "ardia*") 
        
               ), "We expected no PEX or scie leaked in the CWD."

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scie builds, --output, and resulting artifacts #2516

{{title}}

Replies: 2 comments 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Scie builds, --output, and resulting artifacts #2516

martina-oefelein Sep 4, 2024

Replies: 2 comments · 6 replies

martina-oefelein Sep 4, 2024 Author

jsirois Sep 4, 2024 Maintainer

Some examples:

jsirois Sep 4, 2024 Maintainer

martina-oefelein Sep 5, 2024 Author

jsirois Sep 9, 2024 Maintainer

jsirois Sep 10, 2024 Maintainer

jsirois Sep 13, 2024 Maintainer

martina-oefelein
Sep 4, 2024

Replies: 2 comments 6 replies

martina-oefelein
Sep 4, 2024
Author

jsirois
Sep 4, 2024
Maintainer

jsirois Sep 4, 2024
Maintainer

martina-oefelein Sep 5, 2024
Author

jsirois Sep 9, 2024
Maintainer

jsirois Sep 10, 2024
Maintainer

jsirois Sep 13, 2024
Maintainer