Generated Pex File works on some hosts, not others

[ghost]

Hi All,

I've created a Pex file that works when run within a running Podman container (hosted on RedHat 8) that I do my development work in, but when I try and run it on a proper VM (either RedHat 7 or 8), it fails to find the bundled python dependencies (it complains about typer in this case, that's just the first that it tries to load).

ModuleNotFoundError: No module named 'typer'

I ran with the verbose flag on all instances, and didn't see any glaring differences when using vimdiff, other then the addresses that libraries were being loaded into or from.

I've generated the pexfile on both the RedHat 7 vm and the Podman container and the same result occurs.

I'm using a script to handle the generation so that it's repeatable.

How can I debug this? Has anyone else seen this?

It's worth noting that in the container it's running as root (rootless podman) but outside it's running as a standard user (and that's how it'd be deployed once working). I've examined the differences in the user environments and have found nothing Python-related that exists and/or is different.

Thanks for any suggestions!

[jsirois]

PEX files don't magically work anywhere. Like anything else you need to be deliberate about enumerating the target environments you want the PEX you build to run on. Towards that end, can you provide the Pex command line used to build the PEX file and the list of target Python interpreters / systems you expect it to run on? For Linux targets the version of glibc available is critical as it plays into manylinux wheel tags.

[ghost]

Thanks for responding!

The command is the same for all environments:

pex --no-pypi --index-url https://my.employer.pypi.com/simple --requirement /tmp/generated-pex-requirements.txt --output-file ./bin/my-product.pex --cert /path/to/my/ca-bundle.crt --jobs 4 --python-shebang /path/to/internal/python --pex-root /path/to/dir/made/by/mktemp

I believe that the internal python interpreter is different through some magic for Redhat7 & 8 (it's hosted on AFS).

Thanks for telling me that pex files aren't expected to be portable across OS versions - that was naive of me!

But: given that It won't work on a RedHat 7 host, even when the Pex file is generated on a 7 host... that's what I find most odd. I'd be completely happy dropping Redhat 8 support for the time being & later having our CI/CD generate a separate pex for each OS. Let's pretend that I never mentioned anything other than Redhat 7 :-)

[jsirois]

Ok, that is a pretty naive command as it stands. You nowhere nail down the Python(s) you're building for. That is done with --python or --complete-platform (--platform is a lossy version of --complete-platform that should probably be avoided). So, what you get now is the PEX will only guaranteed-work for the Python that hosts the pex console script on then machine you build the PEX file on (and the associated glibc version).

To see more what is going on try unzip -qc ./bin/my-product.pex PEX-INFO | jq . (or build your PEX file with --include-tools or --venv and use PEX_TOOLS=1 ./bin/my-product.pex info --indent 2).

You'll see a map of distributions that represent all the ~wheels your PEX file carries. If you pass multiple --python or --complete-platform or a mixture, you'll likely see many variants for a given platform-specific wheel to satisfy each of those. At runtime, just the right subset is activated. The activation works by matching wheel tags (and looking at Requires-Python wheel METADATA as well as environment markers).

If you're not familiar with wheel tag matching, see these PEPs for a start:

• https://peps.python.org/pep-0425/
• https://peps.python.org/pep-0600/

For Requires-Python see: https://peps.python.org/pep-0345/#requires-python

For environment markers see: https://peps.python.org/pep-0508/#environment-markers

[jsirois]

As an aside, --python-shebang /path/to/internal/python is pretty restrictive - that means the target hosts must all have the python you expect at that exact path. If the platform specific wheels in the PEX are to work, the array of hosts with Python at /path/to/internal/python must also have glibc compatible with all your platform-specific wheels (more on that in the https://peps.python.org/pep-0600/ PEP linked above). My naive guess is its this latter glibc incompatibility that may be causing you a problem, but you've provided very litte actionable debugging information. That you could gather by running the PEX with PEX_VERBOSE=9 ./bin/my-product.pex ... and reporting the log with redactions as needed.