Pyinstaller not working with manylinux2014 because libpython3.so missing

[ikus060]

Description of the issue

Context information (for bug reports)

• Output of pyinstaller --version: 4.4
• Version of Python: 3.7
• Platform: manylinux
• How you installed Python: N/A
• Did you also try this on another platform? Does it work there?
• try the latest development version, using the following command:
• follow all the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

Make sure everything is packaged correctly

• start with clean installation
• use the latest development version
• Run your frozen program from a command window (shell) — instead of double-clicking on it
• Package your program in --onedir mode
• Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
• Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

Stacktrace / full error message

$ docker run -it -v `pwd`:/build quay.io/pypa/manylinux2014_x86_64
[root@0a17017e3ac9 /]# 
[root@0a17017e3ac9 /]# 
[root@0a17017e3ac9 /]# /opt/
_internal/ python/    rh/        
[root@0a17017e3ac9 /]# /opt/python/
cp310-cp310/      cp36-cp36m/       cp37-cp37m/       cp38-cp38/        cp39-cp39/        pp37-pypy37_pp73/
[root@0a17017e3ac9 /]# /opt/python/cp37-cp37m/
bin/     include/ lib/     share/   
[root@0a17017e3ac9 /]# /opt/python/cp37-cp37m/
bin/     include/ lib/     share/   
[root@0a17017e3ac9 /]# /opt/python/cp37-cp37m/bin/
2to3               pip                pydoc3.7           python3.7          python3-config     
2to3-3.7           pip3               pyproject-build    python3.7-config   pyvenv             
idle3              pip3.7             python             python3.7m         pyvenv-3.7         
idle3.7            pydoc3             python3            python3.7m-config  wheel              
[root@0a17017e3ac9 /]# cd /opt/python/cp37-cp37m/bin/
[root@0a17017e3ac9 bin]# 
[root@0a17017e3ac9 bin]# 
[root@0a17017e3ac9 bin]# 
[root@0a17017e3ac9 bin]# 
[root@0a17017e3ac9 bin]# 
[root@0a17017e3ac9 bin]# ./pip install pyinstaller
Collecting pyinstaller
  Downloading pyinstaller-4.4-py3-none-manylinux2014_x86_64.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 2.7 MB/s 
Requirement already satisfied: importlib-metadata in /opt/_internal/cpython-3.7.11/lib/python3.7/site-packages (from pyinstaller) (4.6.1)
Requirement already satisfied: setuptools in /opt/_internal/cpython-3.7.11/lib/python3.7/site-packages (from pyinstaller) (57.4.0)
Collecting altgraph
  Downloading altgraph-0.17-py2.py3-none-any.whl (21 kB)
Collecting pyinstaller-hooks-contrib>=2020.6
  Downloading pyinstaller_hooks_contrib-2021.2-py2.py3-none-any.whl (190 kB)
     |████████████████████████████████| 190 kB 22.5 MB/s 
Requirement already satisfied: zipp>=0.5 in /opt/_internal/cpython-3.7.11/lib/python3.7/site-packages (from importlib-metadata->pyinstaller) (3.5.0)
Requirement already satisfied: typing-extensions>=3.6.4 in /opt/_internal/cpython-3.7.11/lib/python3.7/site-packages (from importlib-metadata->pyinstaller) (3.10.0.0)
Installing collected packages: pyinstaller-hooks-contrib, altgraph, pyinstaller
Successfully installed altgraph-0.17 pyinstaller-4.4 pyinstaller-hooks-contrib-2021.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[root@0a17017e3ac9 bin]# echo "print('coucou')" > ~/main.py
[root@0a17017e3ac9 bin]# 
[root@0a17017e3ac9 bin]# ./pyinstaller ~/main.py
52 INFO: PyInstaller: 4.4
52 INFO: Python: 3.7.11
53 INFO: Platform: Linux-5.11.8-051108-generic-x86_64-with-centos-7.9.2009-Core
53 INFO: wrote /opt/_internal/cpython-3.7.11/bin/main.spec
56 INFO: UPX is not available.
58 INFO: Extending PYTHONPATH with paths
['/root', '/opt/_internal/cpython-3.7.11/bin']
62 INFO: checking Analysis
62 INFO: Building Analysis because Analysis-00.toc is non existent
62 INFO: Initializing module dependency graph...
64 INFO: Caching module graph hooks...
72 INFO: Analyzing base_library.zip ...
2086 INFO: Processing pre-find module path hook distutils from '/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
2086 INFO: distutils: retargeting to non-venv dir '/opt/_internal/cpython-3.7.11/lib/python3.7'
3279 INFO: Caching module dependency graph...
3429 INFO: running Analysis Analysis-00.toc
3444 INFO: Analyzing /root/main.py
3446 INFO: Processing module hooks...
3447 INFO: Loading module hook 'hook-encodings.py' from '/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/hooks'...
3520 INFO: Loading module hook 'hook-sysconfig.py' from '/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/hooks'...
3529 INFO: Loading module hook 'hook-heapq.py' from '/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/hooks'...
3530 INFO: Loading module hook 'hook-xml.py' from '/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/hooks'...
3798 INFO: Loading module hook 'hook-difflib.py' from '/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/hooks'...
3799 INFO: Loading module hook 'hook-distutils.py' from '/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/hooks'...
3800 INFO: Loading module hook 'hook-pickle.py' from '/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/hooks'...
3811 INFO: Looking for ctypes DLLs
3811 INFO: Analyzing run-time hooks ...
3814 INFO: Including run-time hook '/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py'
3817 INFO: Including run-time hook '/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/hooks/rthooks/pyi_rth_inspect.py'
3823 INFO: Looking for dynamic libraries
4180 INFO: Looking for eggs
4180 INFO: Python library not in binary dependencies. Doing additional searching...
Traceback (most recent call last):
  File "./pyinstaller", line 8, in <module>
    sys.exit(run())
  File "/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/__main__.py", line 126, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 758, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 705, in build
    exec(code, spec_namespace)
  File "/opt/_internal/cpython-3.7.11/bin/main.spec", line 19, in <module>
    noarchive=False)
  File "/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 254, in __init__
    self.__postinit__()
  File "/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/building/datastruct.py", line 159, in __postinit__
    self.assemble()
  File "/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 496, in assemble
    self._check_python_library(self.binaries)
  File "/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 601, in _check_python_library
    python_lib = bindepend.get_python_library_path()
  File "/opt/_internal/cpython-3.7.11/lib/python3.7/site-packages/PyInstaller/depend/bindepend.py", line 959, in get_python_library_path
    raise IOError(msg)
OSError: Python library not found: libpython3.7.so, libpython3.7mu.so.1.0, libpython3.7m.so, libpython3.7.so.1.0, libpython3.7m.so.1.0
    This would mean your Python installation doesn't come with proper library files.
    This usually happens by missing development package, or unsuitable build parameters of Python installation.

    * On Debian/Ubuntu, you would need to install Python development packages
      * apt-get install python3-dev
      * apt-get install python-dev
    * If you're building Python by yourself, please rebuild your Python with `--enable-shared` (or, `--enable-framework` on Darwin)

[rokm]

PyInstaller requires libpython3.x.so shared library, as it dynamically loads core Python API functions from it (e.g., to set up embedded python interpreter). Therefore, it cannot work with python that was not built with --enable-shared (or --enable-framework on macOS), which seems to be the case for pythons in this container.

[ikus060]

Could we then re-open this bug and provide documentation about how to generate an executable that is compatible with many Linux distribution ? manylinux docker image is kinda the defacto way to provide wheel package compatible for all Linux.

Would love to get more information how to create such binary with pyinstaller.

[bwoodsend]

I can sympathise. I really wish Python had never gone down the path of merging libpython.so with the python executable - it leaves pretty much every Python extension module linked via a symbol hack because there's nothing to link against.

You'd have to either compile Python from source on the docker image or install Python via the native package manager (yum add python3-dev I think) - in which case you won't get the choice of Python version but it'll be linked against the old version of glibc on the docker image so that it'll still work on most Linuxs. In either case, you'll probably want to use the centos7 base image directly rather than using manylinux2014_x86_64 as the base image as you won't be using the Pythons installed on there.

Alternatively, there is cdrx/docker-pyinstaller which would do the job too.

[ikus060]

@bwoodsend Thanks for the additional information provided. I tried cdrx/docker-pyinstaller but it's a bit too old (updated 2 year ago). More recent is obenn/pyinstaller-linux:python3

[ikus060]

Leaving this here for more details.

manylinux images could be used with pyinstaller.

docker run -it quay.io/pypa/manylinux2014_x86_64 bash
yum install -y python3-devel
pip3 install pyinstaller
pyinstaller main.py