Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker: Add ability to build shared library flavours of CPython #1185

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

rcoup
Copy link

@rcoup rcoup commented Sep 15, 2021

Add a build option to enable building shared/libpython-enabled Python. Helps a number of use-cases (eg: #1149 #793 #255 #30). In mine, using PyInstaller for creating a works-on-many-linux distribution.

Obviously as described in some of the above tickets, it doesn't fit the manylinux requirements and shouldn't be on by default. But adding the option helps people reuse the huge work put into manylinux images without maintaining forks or redoing it all.

Enable via:

MANYLINUX_BUILD_EXTRA="--build-arg PY_SHARED=1" \
PLATFORM=$(uname -m) POLICY=manylinux2014 COMMIT_SHA=latest \
./build.sh

If set, builds both static & shared versions of CPython interpreters into the images.

Shared versions end up in (eg) /opt/python/cp37-cp37m-shared/ alongside the existing static /opt/python/cp37-cp37m/, with a Python binary as /usr/local/bin/python3.7-shared.

@auvipy auvipy requested a review from mayeut September 15, 2021 14:09
@rcoup
Copy link
Author

rcoup commented Sep 15, 2021

If people are happy enough I will update the README/etc too. And could even expose it as a slightly less hidden top-level PY_SHARED=1 option.

rcoup added a commit to koordinates/kart that referenced this pull request Sep 15, 2021
Can't use the vanilla manylinux images because CPython isn't built as shared.
koordinates2/manylinux2014-pyshared images are a minor tweak.
See pypa/manylinux#1185 for details.

After that just install PyInstaller into the image before running.

Manylinux2014-derived libraries use libxcrypt to provide libcrypt.so.2, but that
doesn't seem to end up bundled. Do it manually. Probably goes away with a newer
PyInstaller.
rcoup added a commit to koordinates/kart that referenced this pull request Sep 15, 2021
Can't use the vanilla manylinux images because CPython isn't built as shared.
koordinates2/manylinux2014-pyshared images are a minor tweak.
See pypa/manylinux#1185 for details.

After that just install PyInstaller into the image before running.

Manylinux2014-derived libraries use libxcrypt to provide libcrypt.so.2, but that
doesn't seem to end up bundled. Do it manually. Probably goes away with a newer
PyInstaller.
Copy link
Contributor

@auvipy auvipy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me

Enable via:

    PY_SHARED=1 \
    PLATFORM=$(uname -m) POLICY=manylinux2014 COMMIT_SHA=latest \
    ./build.sh

If set, builds both static & shared versions of CPython into the images.
Shared versions end up in (eg) /opt/python/cp37-cp37m-shared/ alongside the
existing static /opt/python/cp37-cp37m/, with a Python binary as
/usr/local/bin/python3.7-shared.
@rcoup
Copy link
Author

rcoup commented Sep 17, 2021

If people are happy enough I will update the README/etc too. And could even expose it as a slightly less hidden top-level PY_SHARED=1 option.

I've done that in the latest push.

Copy link
Member

@mayeut mayeut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking the time to do this.
Before this can be merged, please do explain why you'd need a manylinux like image. The issues you mentioned provides another way to handle the PyInstaller/cx-freeze use-case with minimal effort. What use-case is not solved with a minimal "CentOS" (or other base image) image with python ?
I'd rather understand the added value for downstream repos before including bits of code that will never be tested here.

ABI_TAG=$(${PREFIX}/bin/python ${MY_DIR}/python-tag-abi-tag.py)
ln -s ${PREFIX} /opt/python/${ABI_TAG}
ln -s ${PREFIX} /opt/python/${ABI_TAG}${SUFFIX:-}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ln -s ${PREFIX} /opt/python/${ABI_TAG}${SUFFIX:-}
ln -s ${PREFIX} /opt/python/${ABI_TAG}${SUFFIX}

@termim
Copy link

termim commented Feb 22, 2022

Before this can be merged, please do explain why you'd need a manylinux like image.

To build and test PyInstaller/cx-freeze based executables for every supported python version.

The issues you mentioned provides another way to handle the PyInstaller/cx-freeze use-case with minimal effort.

No, the issues mentioned basically say - use whatever python provided by the base image or build your own. And manylinux project itself proves that it is not a minimal effort.

What use-case is not solved with a minimal "CentOS" (or other base image) image with python ?

The use case is pretty obvious - base your work on top of well done manylinux project and avoid reinventing this wheel (properly build all supported python versions) over and over again.

I'd rather understand the added value for downstream repos before including bits of code that will never be tested here.

The added value for downstream repos is ability to use properly built all supported python versions with up to date supporting libraries like openssl etc. for building/testing executables from python scripts.

@mayeut
Copy link
Member

mayeut commented Feb 26, 2022

To build and test PyInstaller/cx-freeze based executables for every supported python version.

@termim, can you please explain the use-case (or better, provide a real reference for the use-case) of building an application/executable with every supported python version ? This is where I'm lost. If you're building an application, only one python version should be enough ?

Not sure if it's been mentioned before but there's also https://github.com/indygreg/python-build-standalone/ python builds which are meant to be highly portable builds of python (manylinux just build without any care that the python builds themselves should be portable, only the built wheels are taken care of).

@termim
Copy link

termim commented Feb 26, 2022

To build and test PyInstaller/cx-freeze based executables for every supported python version.

@termim, can you please explain the use-case (or better, provide a real reference for the use-case) of building an application/executable with every supported python version ? This is where I'm lost. If you're building an application, only one python version should be enough ?

Example: my application uses Python-3.8 and I build an executable with PyInstaller-4.8 currently. Obviously I plan to update to more recent Python, PyInstaller and a bunch of other packages my application uses. Obviously I have to test it as an executable on different Linux distros as well. Obviously I need to properly build Python for that. Currently I have to maintain myself these builds. Manylinux project does exactly that except one little option - build shared libpython. This option does not need to be on by default or there could be 'shared' flavor of x86_64 images - in any case that would help a lot to those who use PyInstaller/cx-Freeze. That would be very natural extension to manylinux project IMHO.

Not sure if it's been mentioned before but there's also https://github.com/indygreg/python-build-standalone/ python builds which are meant to be highly portable builds of python (manylinux just build without any care that the python builds themselves should be portable, only the built wheels are taken care of).

There is a lot of duplication in this project and manylinux. IMHO manylinux does better job by explicitly addressing different glibc versions and providing well defined policies and I do not see why python built with it should not be as portable as the wheels.

@black-sliver
Copy link

black-sliver commented Mar 27, 2022

👍 for /opt/python/cp39-cp39-shared.
My use-case would be that the project I want to bundle in an AppImage uses (and kind of depends on) cx_Freeze and is going to move to py3.9+ soon. I chose the AppImage path specifically because it would allow users to run it on older OS with older glibc that do not ship the latest and greatest python and building that that on a manylinux build would give me a defined glibc dependency that is equal to the wheel's.

@pkit
Copy link

pkit commented Mar 7, 2023

Python community is notoriously sloppy with how glibc dependencies are handled.
This project is the only place where some sanity is preserved.
Not having libpython.so though is a major dealbreaker for people who need a stable python environment, with clearly defined glibc dependency.
"Do it yourself" is not a good idea, because in the end it will be yet another manylinux just less used/tested.
You can look into aws-cli for amount of hoops people need to jump.

@pkit
Copy link

pkit commented Mar 7, 2023

@mayeut

This is where I'm lost. If you're building an application, only one python version should be enough ?

Mwahahaha. We are not in the fantasy world with rainbows and unicorns. We live in the harsh reality where some library update may require a new python version. Now what?

marchelbling added a commit to marchelbling/manylinux that referenced this pull request Aug 28, 2024
**Summary:**
See pypa#1185

We propose to:
* update the branch and "bake" the shared libraries build
* remove old Python (<3.8) versions that we do not care
* remove the GIL build
marchelbling added a commit to marchelbling/manylinux that referenced this pull request Aug 28, 2024
**Summary:**
See pypa#1185

We propose to:
* update the branch and "bake" the shared libraries build
* remove old Python (<3.8) versions that we do not care
* remove the GIL build
* remove compression for static libs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants