Cannot use pex with statically linked python

[zmanji]

due to pypa/pip#6543 it's not possible for pex to resolve dependencies with a statically linked python.

To reproduce in a clean chroot.

$ wget https://github.com/indygreg/python-build-standalone/releases/download/20221220/cpython-3.10.9+20221220-x86_64_v3-unknown-linux-musl-install_only.tar.gz
$ tar xvzf cpython-3.10.9+20221220-x86_64_v3-unknown-linux-musl-install_only.tar.gz 
$ wget https://github.com/pantsbuild/pex/releases/download/v2.1.119/pex
$ ./python/bin/python3 ./pex 'fortune' -o fortune.pex
pid 31 -> /root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/bin/python -sE /root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /root/.pex/pip_cache download --dest /root/.pex/downloads/resolver_download.oodws27t/python.bin.python3.10 fortune --index-url https://pypi.org/simple --retries 5 --timeout 15 exited with 2 and STDERR:
ERROR: Exception:
Traceback (most recent call last):
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/commands/download.py", line 92, in run
    session = self.get_default_session(options)
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 78, in get_default_session
    self._session = self.enter_context(self._build_session(options))
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 88, in _build_session
    session = PipSession(
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/network/session.py", line 248, in __init__
    self.headers["User-Agent"] = user_agent()
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/network/session.py", line 135, in user_agent
    zip(["lib", "version"], libc_ver()),
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 94, in libc_ver
    glibc_version = glibc_version_string()
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 18, in glibc_version_string
    return glibc_version_string_confstr() or glibc_version_string_ctypes()
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 52, in glibc_version_string_ctypes
    process_namespace = ctypes.CDLL(None)
  File "/python/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Dynamic loading not supported

Changing the pip version does not work because the vendored pip is used to fetch the newer pip. I see two possible solutions:

• pex3 is released and vendors a newer pip.
• two pips are vendored much like how two packaging versions are vendored, and when a statically linked python is detected we use the newer pip.

[jsirois]

Two pips is getting big, I'm partial to Pex 3.x vendoring a single newer Pip.

That said, you're about to run into more problems anyhow using Python Build Standalone - those won't be able to build native wheels from sdists - sysconfig has many key bad paths embedded from the PBS build process.

[zmanji]

I forgot to say, this is not an urgent/important thing. Just reporting it to share the news. I agree that PBS is not great for building wheels and fixing this is sure to reveal more issues.

Maybe it would be good just to tag this with a 'pex3' tag, and the issue can be revisited whenever pex3 is nearing release.

[jsirois]

@zmanji - I'm re-reading this issue on a quest to re-look at all outstanding issues you've filed. They tend to be good ones.

due to pypa/pip#6543 it's not possible for pex to resolve dependencies with a statically linked python.

AFAICT that linked issue is a red-herring since the fix was in in the Pip 19 series (and Pex only uses 20.3.4 vendored and newer). FWICT that fix tries to os.confstr("CS_GNU_LIBC_VERSION") to get the glibc version before falling back to ctypes. Since a musl libc system won't have that conf string, everything is still broken at the Pip level (if Pip exists in a running state on musl libc systems it seems to me it must be a patched version of it - and I know the one IndyGreg ships is in-fact patched).

I just tried with https://github.com/indygreg/python-build-standalone/releases/download/20240713/cpython-3.10.14+20240713-x86_64-unknown-linux-musl-install_only.tar.gz and get the same backtrace:

./python/bin/python3 ./pex 'fortune' -o fortune.pex
pid 268587 -> /home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/bin/python -sE /home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/jsirois/.pex/pip/20.3.4-patched/pip_cache download --dest /home/jsirois/.pex/downloads/resolver_download.c6knb953/tmp.issues-2017.python.bin.python3.10 fortune --retries 5 --timeout 15 exited with 2 and STDERR:
ERROR: Exception:
Traceback (most recent call last):
  File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/commands/download.py", line 92, in run
    session = self.get_default_session(options)
  File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 78, in get_default_session
    self._session = self.enter_context(self._build_session(options))
  File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 88, in _build_session
    session = PipSession(
  File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/network/session.py", line 248, in __init__
    self.headers["User-Agent"] = user_agent()
  File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/network/session.py", line 135, in user_agent
    zip(["lib", "version"], libc_ver()),
  File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 94, in libc_ver
    glibc_version = glibc_version_string()
  File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 18, in glibc_version_string
    return glibc_version_string_confstr() or glibc_version_string_ctypes()
  File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 52, in glibc_version_string_ctypes
    process_namespace = ctypes.CDLL(None)
  File "/tmp/issues-2017/python/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Dynamic loading not supported

The real fix only landed a few weeks ago afaict and is unreleased (will be released in 24.2 - probaly next weekend): pypa/pip#12716
I'll see if I can pull in the glibc patches to the vendored Pip Pex uses, which should allow bootstrapping the latest Pip with Charlie's fix on musl libc systems.

[jsirois]

Ok, That picked cleanly, although I had to remove a Python3.5+-style type annotation so this still works with Python 2.7: pex-tool/pip#13

[jsirois]

Ok, that does the trick for the OP. You can have your fortune read by Pex using Pip 20.3.4+patched on musl libc.

[jsirois]

Alright - the fix for this is released in 2.12.0:

:; wget https://github.com/indygreg/python-build-standalone/releases/download/20221220/cpython-3.10.9+20221220-x86_64_v3-unknown-linux-musl-install_only.tar.gz
:; tar xvzf cpython-3.10.9+20221220-x86_64_v3-unknown-linux-musl-install_only.tar.gz
:; wget https://github.com/pantsbuild/pex/releases/download/v2.12.0/pex
:; ./fortune.pex /usr/share/games/fortunes/fortunes
You will stop at nothing to reach your objective, but only because your
brakes are defective.