Constrain the matrix of shebang, platforms and ICs as much as possible. #1540
Comments
Lightly related, but a thing I've wondered about: how awkward would it be to replace the bootstrap/ |
|
What cases that aren't currently handled would be your goal to handle with such a bash header? It does seem that
|
|
That sounds right, although I suppose that I hadn't considered that both steps 1 and 2 would be necessary (implementing interpreter constraints in bash would be a pain). The goal would mostly be to remove a dimension from the description, since |
Yeah - I think the error here is auto-selecting a hashbang. Since these aren't in fact universal, Pex should probably force you to pick one to make it clear you should think about this and know the right answer for your fleet. There are exactly 2 cases I'm aware of where users can't answer this because the fleet is not theirs - the Pex PEX and Pants' PEX. For the former, Pants uses your suggestion (externalized though) to find the bootstrap Python to run the Pex PEX with. |
|
I realized step 1 in this list above could be done in Python code that emits a bash header. That simplifies the bash greatly. I still think that Pex needs to keep its Python 1st tradition to both not break current users and to make sure PEX by default can ship to a machine with Python and work - it would be bad to land on a machine with Python in the PATH but not a shell and fail to run. It would also hinder the many-times-stalled effort to run out of the box on Windows. So, if put behind a flag though, this is viable. In fact it cleans up a whole mess of code over in Pants designed to avoid ~50ms of leftover overhead in the PEX zip figuring out it has an already installed Numbers look good. Does not affect the cold case but is snappy for the Cold: Seed: Warm: N.B.: I threw in direct execution of the venv pex script in the warm perf comparison. So we lose ~2ms by not executing the venv pex script directly. This is the smae penalty Pants experiences today - it just writes its own bash re-directory script instead of using one embedded in the PEX header like this. The interpreter selection for $ head -71 cowsay.bash.venv.pex
#!/usr/bin/env sh
# N.B.: This script should stick to syntax defined for POSIX `sh` and
# avoid non-builtins:
# https://pubs.opengroup.org/onlinepubs/9699919799/idx/shell.html
set -eu
VENV="1"
# N.B.: This ensures tilde-expansion of the DEFAULT_PEX_ROOT value.
DEFAULT_PEX_ROOT="$(echo ~/.pex)"
PEX_ROOT="${PEX_ROOT:-${DEFAULT_PEX_ROOT}}"
PEX="${PEX_ROOT}/venvs/346aee797b51ee3f468179886b3a5db7c6d723a3/0c2af63c3815d1d03077ee9c1f2cbc64e6c7925d/pex"
if [ -n "${VENV}" -a -x "${PEX}" ]; then
exec "${PEX}" "$@"
fi
find_python() {
for python in \
"python3.10" \
"python2.7" \
"python3.5" \
"python3.6" \
"python3.7" \
"python3.8" \
"python3.9" \
"python3.11" \
"python3" \
"python2" \
"python" \
; do
if command -v "${python}" 2>/dev/null; then
return
fi
done
}
python_exe="$(find_python)"
if [ -n "${python_exe}" ]; then
if [ -z "${VENV}" -a -e "${PEX}" ]; then
exec "${python_exe}" "${PEX}" "$@"
else
# The slow path, run the PEX zipapp so it can rebuild its fast
# path layout under the PEX_ROOT.
exec "${python_exe}" "$0" "$@"
fi
else
echo >&2 "Failed to find any of these python binaries on the PATH:"
for python in \
"python3.10" \
"python2.7" \
"python3.5" \
"python3.6" \
"python3.7" \
"python3.8" \
"python3.9" \
"python3.11" \
"python3" \
"python2" \
"python" \
; do
echo >&2 "${python}"
done
echo >&2 "Either adjust your $PATH which is currently:"
echo >&2 "${PATH}"
echo >&2 -n "Or else install an appropriate Python that provides "
echo >&2 "one of the binaries in this list."
exit 1
fi
P!
.bootstrap/P!.bootstrap/pex/P!�}; |
|
And busybox crushes all comers: |
|
Super awesome. In the spirit of making progress on this ticket (and understanding that what you said about wanting to support environments without shells installed means that python-shebang will need to stick around as an option), it seems like maybe |
|
Pants can definitely choose that as a default for its users. Pex cannot until the Pex 3 release - my policy is to only fix bugs and add features but not break users without bumping major. |
Although its not always possible to derive an appropriate shebang that will work for multiplatform PEXes, we now do so when possible and warn when we cannot. Fixes pex-tool#1540
Although its not always possible to derive an appropriate shebang that will work for multiplatform PEXes, we now do so when possible and warn when we cannot. Fixes pex-tool#1540
Although its not always possible to derive an appropriate shebang that will work for multi-platform PEXes, we now do so when possible and warn when we cannot. Fixes pex-tool#1540
Although it's not always possible to derive an appropriate shebang that will work for multi-platform PEXes, we now do so when possible and warn when we cannot. Fixes #1540
Building and booting a PEX can involve the interplay of 4 concepts currently:
These all need to be in alignment to produce a PEX that "boots" properly. Further, they need to be particular beyond that in order to squeeze performance out of some combinations of those items.
Pex cannot ensure alignment, let along maximally performing alignment, in all cases. It can probably catch a few out-of-aligment cases though and - preferably - automatically bring them in alignment, or - less preferably - warn or potential issues. For the latter, care will be needed not to provided unwanted nannying for a knowledgeable user or a too sharp knife (the current situation) for the casual user.
An example of automated alignment would be basing the default shebang for a single --platform PEX, or else multiplatform PEX where all platforms share the same major / minor versions in their platform tag, off that major / minor version pair. I.E.: for
--platform linux-x86_64-cp-37-cp37m --platform macosx-10.13-x86_64-cp-37-muse a default shebang of#!/usr/bin/env python3.7. This is in contrast to the behavior today, which is the shebang defaulting to#!/usr/bin/env pythonX.Ywhere X / Y are the major / minor versions of the interpreter used to run the Pex CLI, and which need bear no relation to the platforms selected.The answer to what can be and should be constrained will change once #1020 is implemented, but even shy of that the status quo can be improved to prevent this style of unambiguous mis-aligment.
The text was updated successfully, but these errors were encountered: