How to create platform portable wheels (manylinux) ? #329
Comments
|
@tristan0x @mgeplf : you might have more experience on this (I am doing this first time). Do you have any suggestion? |
|
|
|
Are you sure the python installation is manylinux compatible? def is_manylinux1_compatible():
# Only Linux, and only x86-64 / i686
from distutils.util import get_platform
if get_platform() not in ["linux-x86_64", "linux-i686"]:
return False
# Check for presence of _manylinux module
try:
import _manylinux
return bool(_manylinux.manylinux1_compatible)
except (ImportError, AttributeError):
# Fall through to heuristic check below
pass
# Check glibc version. CentOS 5 uses glibc 2.5.
return have_compatible_glibc(2, 5) |
|
Do you happen to have the logs from auditwheel? |
I would recommend using the |
|
@pramodk: I just had a quick look, if I reduce the complexity, by: c) Using the manywheels python version: I get this output: This appears to be because the path to the libraries isn't being passed properly (ie: |
|
DYNAMIC=OFF (the default) is intended to mean that the nrnpython interface is linked into libnrniv.so and |
|
Ok, good to know. The method that the above is handled may need to change, as per the documentation of manylinux (https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1): |
|
One more aspect of DYNAMIC=OFF is perhaps relevant. I.e. when nrniv is launched, then the |
|
Thank you very much @mgeplf and @tristan0x for comments. I am bit late here but here are answers/details:
I used method provided here and all versions seems manylinux compatible:
The reason that I shifted from But I see the point of why libraries are not there in first place.
I see that the python library paths are incomplete: We are using module from pybond11 to find python library paths and it is / was quite stable except in this manylinux platform where things are incomplete (?). I also tried autoconf build to see if it detect python libraries correctly if python is used from After reading Vtk and OpenCV related posts, I see that the common practice (suggestion) is With this, I am able to build neuron with python from I still have to test these builds and see if wheels are usable on different platforms (will do this tomorrow). |
|
Using I uploaded new wheel to test.pypi but see same error while installing those on centos: So there is still something else. @mgeplf : In NSE stack, are you guys building wheels using manylinux platform? |
|
Here is another update: Tried upgrading pip: And then tried following which seems to work: but I still need to dig into: # python3.6
Python 3.6.8 (default, Aug 7 2019, 17:28:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from neuron import h
Traceback (most recent call last):
File "/usr/local/lib64/python3.6/site-packages/neuron/__init__.py", line 113, in <module>
import hoc
ModuleNotFoundError: No module named 'hoc'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib64/python3.6/site-packages/neuron/__init__.py", line 117, in <module>
import neuron.hoc
ImportError: /usr/local/lib64/python3.6/site-packages/neuron/hoc.cpython-36m-x86_64-linux-gnu.so: undefined symbol: nrnpy_site_problem
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib64/python3.6/site-packages/neuron/__init__.py", line 119, in <module>
exec("import neuron.hoc%d%d as hoc" % (sys.version_info[0], sys.version_info[1]))
File "<string>", line 1, in <module>Will do couple of more tests and will update here. |
I have been using the manylinux1. We also transitioned from building them 'custom' (like above), to using the multibuild For example: https://github.com/BlueBrain/libsonata-wheels From my experience, I would work out the bugs to make sure you can build the wheels by hand before moving to multibuild - it cleans things up, but also makes it harder to debug. Can you post the output of Also, can you make sure that |
|
Note; the traceback you posted here: Is trying to do a |
|
I have resolved most of the issues but what is making bit problematic is link to other shared libraries like readline, ncurses etc from neuron.
obviously because we haven't linked to readline. If I pre-load (just for testing) then it works: So in principal there is no issue with the wheel itself. Linking to readline/ncurses libraries lead to other errors like this one. If I look into generate wheel, indeed there those .libs directories: I will pass by your desk tomorrow or Monday. |
|
Is readline really a requirement? Again, it would be helpful to see what Come by whenever, I'm busy until lunch, and Monday's are usually more busy than Fridays, but whatever works for you. |
|
Huge thanks to @mgeplf and @ferdonline for debugging this issue of In summary: when nrniv is included into wheel, auditwheel detects that binary file is linked to libnrniv and other libraries and create additional In the latest commit I disabled building nrniv binary for now. One of the suggestion was to write python shim for nrniv to avoid such issues for portable issues without workarounds. Renaming linux wheel as manylinux works. But after trying wheel built on manylinux onto centos, using Here is docker image: FROM quay.io/pypa/manylinux2010_x86_64
MAINTAINER Pramod Kumbhar <[email protected]>
# Need to install sudo first:
RUN yum install -y sudo
# Can only run sudo from a terminal on CentOS 5.
# While we are still root, install the necessary dependencies for Malmo:
RUN yum -y install \
git \
swig \
doxygen \
libxslt \
gcc-c++ \
bzip2-devel \
python34-tkinter \
wget \
software-properties-common \
xpra \
libgl1-mesa-dri \
make \
vim \
curl \
gfortran \
unzip \
bison \
flex \
pkg-config \
autoconf \
automake \
make \
openssh-server \
libopenmpi-dev \
libhdf5-serial-dev \
libtool \
zlib-devel \
cmake3 \
ncurses-devel.x86_64 \
gcc \
patch \
make
RUN sudo yum -y install cmake3 ncurses-devel.x86_64 gcc patch make
ENV PATH ${HOME}/.local/bin:$PATH
ENV PATH /opt/python/cp27-cp27m/bin:/opt/python/cp27-cp27mu/bin:/opt/python/cp35-cp35m/bin:/opt/python/cp36-cp36m/bin:/opt/python/cp37-cp37m/bin:$PATH
# default software required
RUN python2.7 -m pip install --upgrade --user scipy matplotlib bokeh ipython cython pytest
RUN python3.5 -m pip install --upgrade --user scipy matplotlib bokeh ipython cython pytest
RUN python3.6 -m pip install --upgrade --user scipy matplotlib bokeh ipython cython pytest
RUN python3.7 -m pip install --upgrade --user scipy matplotlib bokeh ipython cython pytest
# for wheel
RUN python2.7 -m pip install --upgrade --user twine auditwheel pip
RUN python3.5 -m pip install --upgrade --user twine auditwheel pip
RUN python3.6 -m pip install --upgrade --user twine auditwheel pip
RUN python3.7 -m pip install --upgrade --user twine auditwheel pip
RUN git clone https://github.com/neuronsimulator/nrn.git -b setuppy neuron-yale
WORKDIR ${HOME}/neuron-yale
RUN git checkout setuppy
RUN git pull
RUN python2.7 setup.py bdist_wheel
RUN python3.5 setup.py bdist_wheel
RUN python3.6 setup.py bdist_wheel
RUN python3.7 setup.py bdist_wheel
# repair wheel? see : https://malramsay.com/post/perils_of_packaging/
RUN LD_LIBRARY_PATH=`pwd`/_install2717/lib python3.6 -m auditwheel repair dist/NEURON-7.8.11-cp27-cp27m-linux_x86_64.whl
RUN LD_LIBRARY_PATH=`pwd`/_install359/lib python3.6 -m auditwheel repair dist/NEURON-7.8.11-cp35-cp35m-linux_x86_64.whl
RUN LD_LIBRARY_PATH=`pwd`/_install369/lib python3.6 -m auditwheel repair dist/NEURON-7.8.11-cp36-cp36m-linux_x86_64.whl
RUN LD_LIBRARY_PATH=`pwd`/_install375/lib python3.6 -m auditwheel repair dist/NEURON-7.8.11-cp37-cp37m-linux_x86_64.whland build image as: And this creates wheels successfully: $ ls -lrt wheelhouse/
total 18172
-rw-r--r-- 1 kumbhar kumbhar 4505433 Nov 29 23:27 NEURON-7.8.11-cp27-cp27m-manylinux2010_x86_64.whl
-rw-r--r-- 1 kumbhar kumbhar 4657387 Nov 29 23:27 NEURON-7.8.11-cp35-cp35m-manylinux2010_x86_64.whl
-rw-r--r-- 1 kumbhar kumbhar 4718151 Nov 29 23:27 NEURON-7.8.11-cp36-cp36m-manylinux2010_x86_64.whl
-rw-r--r-- 1 kumbhar kumbhar 4721144 Nov 29 23:28 NEURON-7.8.11-cp37-cp37m-manylinux2010_x86_64.whl |
|
@nrnhines : here are instructions to re-produce various issue with wheel that we discussed today. All are related to path detection :
This will create wheel in dist:
try with PYTHONHOME
→ ./myenv/bin/nrnivmodl .
/Users/kumbhar/Downloads/tmp/nrn
ls: ./*.mod: No such file or directory
ls: ./*.inc: No such file or directory
make: /Users/kumbhar/Downloads/tmp/nrn/_install374/bin/nrnmech_makefile: No such file or directory
make: *** No rule to make target `/Users/kumbhar/Downloads/tmp/nrn/_install374/bin/nrnmech_makefile'. Stop.with env variables only it works: → NRNHOME=`pwd`/myenv NEURONHOME=`pwd`/myenv/share/nrn ./myenv/bin/nrnivmodl .
/Users/kumbhar/Downloads/tmp/nrn
ls: ./*.mod: No such file or directory
ls: ./*.inc: No such file or directory
-> Compiling mod_func.c
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -O3 -DNDEBUG -I/Users/kumbhar/Downloads/tmp/nrn/myenv/include -fPIC -c mod_func.c -o mod_func.o
=> LINKING library x86_64/libnrnmech.0.0.dylib Mod files:
(cd .. ; /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -O3 -DNDEBUG -DVERSION_INFO='7.8.11' -std=c++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -fPIC -L/usr/local/opt/ruby/lib -I /Users/kumbhar/Downloads/tmp/nrn/myenv/include -o x86_64/libnrnmech.0.0.dylib -Wl,-install_name,@rpath/libnrnmech.0.0.dylib \
x86_64/mod_func.o -L/Users/kumbhar/Downloads/tmp/nrn/myenv/lib -lnrniv -Wl,-rpath,/Users/kumbhar/Downloads/tmp/nrn/myenv/lib -lreadline)
(cd .. ; rm -f x86_64/.libs/libnrnmech.so ; mkdir -p x86_64/.libs ; ln -s ../../x86_64/libnrnmech.0.0.dylib x86_64/.libs/libnrnmech.so)
Successfully created x86_64/specialbut note that CC and CXX compiler paths are hard-coded which needs to be fixed:
→ cat myenv/bin/nrnmech_makefile | grep "CC\|CXX"
CC = /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
CXX = /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++We can simply avoid setting CC and CXX and assume that Makefile will use default C/C++ compiler? |
|
The puzzle that I'm having is that the generates a hoc module that dynamically loads but the ends up with a hoc module that does NOT refer to libnrnpython3.so (it does refer to libnrniv.so) |
|
I had success with |
|
@nrnhines : my guess is that you haven't removed build & installed directories created during creation of wheel? i.e. python3.7 setup.py bdist_wheel
rm -rf build _install*once above directories are removed, you should be able to produce the warnings/errors: |
|
I'm not having a problem whether or not is executed after I'm installing with pip using, in a fresh terminal, This allows successful 'import neuron' with either a launch of python or nrniv. But my issues now are to get NEURONHOME setup correctly without an explicit environment variable and to get nrnpyenv.sh to return the correct values of PYTHONHOME and NRN_PYLIB. I believe that NEURONHOME can be robustly setup (if it is not already an environment variable) with the dlopen strategy. I hope that nrnpyenv.sh can be fixed up to do the right thing when python environments are being used. When this simplest case is working then I think it will be straightforward to have InterViews dynamically loaded and it may be the case that dynamic loading of mpi/python will work as is with the robust NEURONHOME. Note that this is all for linux/mac. Windows is position independent with respect to installing with setup.exe but I believe it makes use of registry variables and I expect that for pip install, it might be able to use the same kind of methods as linux/mac. We'll see... |
|
By the way, I think a lot of improvments are being blocked waiting for merging the .pramodk/cmake_master_merge into master. If we make a major change to that pull request the |
Agree. Taking a look at that right away! |
|
Note from Fernando : On OSX we set following env variable to make wheel compatible with previous OS X releases: |
|
@ferdonline @iomaganaris : This is on BB5 with Red Hat 7.6 and wheel is being built on manylinux1 by Fernando: module load archive/2019-11
module load python
python -m venv env-36
. env-36/bin/activate
pip install dist/NEURON-7.8.1-cp36-cp36m-manylinux1_x86_64.whl
pip install dist/NEURON-7.8.1-cp36-cp36m-manylinux1_x86_64.whl
Processing ./dist/NEURON-7.8.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy>=1.13.1 (from NEURON==7.8.1)
Cache entry deserialization failed, entry ignored
Downloading https://files.pythonhosted.org/packages/07/08/a549ba8b061005bb629b76adc000f3caaaf881028b963c2e18f811c6edc1/numpy-1.18.2-cp36-cp36m-manylinux1_x86_64.whl (20.2MB)
100% |████████████████████████████████| 20.2MB 64kB/s
Installing collected packages: numpy, NEURON
Successfully installed NEURON-7.8.1 numpy-1.18.2All installs fine. And now using it: (env-36) kumbhar@bbpv1:~/tmp$ nrniv
NEURON -- VERSION 7.8.0-147-g21af5339+ python_wheel (21af5339+) 2020-03-27
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits
loading membrane mechanisms from x86_64/.libs/libnrnmech.so
Additional mechanisms from files
mod//halfgap.mod
oc>quit()
(env-36) kumbhar@bbpv1:~/tmp$ nrniv -python
NEURON -- VERSION 7.8.0-147-g21af5339+ python_wheel (21af5339+) 2020-03-27
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits
loading membrane mechanisms from x86_64/.libs/libnrnmech.so
Additional mechanisms from files
mod//halfgap.mod
>>> import neuron
>>>
(env-36) kumbhar@bbpv1:~/tmp$ python
Python 3.6.5 (default, Dec 19 2018, 05:00:45)
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import neuron
>>>
(env-36) kumbhar@bbpv1:~/tmp$ rm -rf x86_64/
(env-36) kumbhar@bbpv1:~/tmp$ nrnivmodl mod/
Creating x86_64 directory for .o files.
/gpfs/bbp.cscs.ch/home/kumbhar/tmp
ls: cannot access mod//*.inc: No such file or directory
mod//halfgap.mod
halfgap.mod
mkdir -p x86_64
-> Compiling mod_func.c
/gpfs/bbp.cscs.ch/apps/hpc/jenkins/deploy/compilers/2018-12-18/linux-rhel7-x86_64/gcc-4.8.5/gcc-6.4.0-i6lyqfscua/bin/gcc -pthread -O2 -I. -I.. -I/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include -fPIC -c mod_func.c -o mod_func.o
-> NMODL halfgap.mod
MODLUNIT=/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/share/nrn/lib/nrnunits.lib \
/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/bin/nocmodl halfgap.mod
Translating halfgap.mod into halfgap.c
Thread Safe
-> Compiling x86_64/halfgap.c
(cd .. ; /gpfs/bbp.cscs.ch/apps/hpc/jenkins/deploy/compilers/2018-12-18/linux-rhel7-x86_64/gcc-4.8.5/gcc-6.4.0-i6lyqfscua/bin/gcc -pthread -O2 -I. -I.. -I/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include -fPIC -c x86_64/halfgap.c -o x86_64/halfgap.o)
=> LINKING library x86_64/libnrnmech.0.0.so Mod files: halfgap.mod
(cd .. ; /gpfs/bbp.cscs.ch/apps/hpc/jenkins/deploy/compilers/2018-12-18/linux-rhel7-x86_64/gcc-4.8.5/gcc-6.4.0-i6lyqfscua/bin/g++ -pthread -O2 -DVERSION_INFO='7.8.1' -std=c++11 -shared -fPIC -I /gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include -o x86_64/libnrnmech.0.0.so -Wl,-soname,libnrnmech.0.0.so \
x86_64/mod_func.o x86_64/halfgap.o -L/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/lib -lnrniv -Wl,-rpath,/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/lib -lreadline)
(cd .. ; rm -f x86_64/.libs/libnrnmech.so ; mkdir -p x86_64/.libs ; ln -s ../../x86_64/libnrnmech.0.0.so x86_64/.libs/libnrnmech.so)
Successfully created x86_64/special
## CHANGING GCC COMPILERS
(env-36) kumbhar@bbpv1:~/tmp$ CC=`which gcc` CXX=`which g++` nrnivmodl mod/
/gpfs/bbp.cscs.ch/home/kumbhar/tmp
ls: cannot access mod//*.inc: No such file or directory
mod//halfgap.mod
halfgap.mod
-> Compiling mod_func.c
/usr/lib64/ccache/gcc -O2 -I. -I.. -I/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include -fPIC -c mod_func.c -o mod_func.o
-> NMODL halfgap.mod
MODLUNIT=/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/share/nrn/lib/nrnunits.lib \
/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/bin/nocmodl halfgap.mod
Translating halfgap.mod into halfgap.c
Thread Safe
-> Compiling x86_64/halfgap.c
(cd .. ; /usr/lib64/ccache/gcc -O2 -I. -I.. -I/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include -fPIC -c x86_64/halfgap.c -o x86_64/halfgap.o)
=> LINKING library x86_64/libnrnmech.0.0.so Mod files: halfgap.mod
(cd .. ; /usr/lib64/ccache/g++ -O2 -DVERSION_INFO='7.8.1' -std=c++11 -shared -fPIC -I /gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include -o x86_64/libnrnmech.0.0.so -Wl,-soname,libnrnmech.0.0.so \
x86_64/mod_func.o x86_64/halfgap.o -L/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/lib -lnrniv -Wl,-rpath,/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/lib -lreadline)
(cd .. ; rm -f x86_64/.libs/libnrnmech.so ; mkdir -p x86_64/.libs ; ln -s ../../x86_64/libnrnmech.0.0.so x86_64/.libs/libnrnmech.so)
Successfully created x86_64/specialThere is one improvement we should do is: i.e. shim for Also, I pushed the Dockefile I have been using in : https://github.com/neuronsimulator/nrn/blob/python_wheel/dist/Dockerfile The Python3.7-manylinux2010 wheel I built is working fine on BB5. But python 3.5 and 3.6 wheels shows errors like: Corresponding non-repaired wheels are installing fine: This means something is missing in Dockerfile? Have to discuss / check this. |
|
Even thought we could move to manulinux2010, manylinux1 is still the very standard. For instance to handle manylinux2010 we need at least pip 19.0.
|
I was using manulinux2010 just because I had docker recipe setup. We can definitely move to newer manylinux1.
that would be ideal!
I don't know much details about this. certainly !
I think NSE have documented something about this. For example here. May be we can extend / combine / inherit? |
|
The newabi3 branch is a work in progress. It will reduce the number of hoc module shared |
There might be some confusion. manylinux1 is the good old (first?) manylinux image, based on centos5. I believe it still receives updates since it seems to be the most widespread form of manylinux, basically old pip's support it, and pretty much any "living" linux. |
Oops, I thought it’s newer version! Anyway, once you build wheel we can test it on different platforms and as long it works, we should be fine! |
|
I pushed wheels for 3.5-3.8 Btw my docker image is: docker.io/ferdonline/manylinux1_steroids_x86_64 |
|
@ferdonline : tested 3.6 and 3.7 and works great on BB5. Could we fix following? You can reproduce by launching |
Testing NEURON wheel : Trail I@ohm314 @jorblancoa @alexsavulescu @iomaganaris @ferdonline @nrnhines Fernando has finished wheel and ready for first testing ⭐ We have wheels created for Python 3.5, 3.6, 3.7 and 3.8. Goal is test on different Linux and OS X distributions (and not only BB5 because we have mostly tested it). So if you have local desktop, laptop, cluster node to test the compatibility, it would be great. How to test?
On Ubuntu, you can have I also saw error @ferdonline @nrnhines : Is this expected? When I installed dev package, everything worked fine: If you see any errors or warning, please report here. |
|
My repository folder is named python_wheel on my linux machine. I have pythons managed by pyenv and the global version is Python 3.7.6. The full output is: |
|
@nrnhines : could you remove “set -e” and run tests again? i.e. run even on errors. I am curious if this is only case with python 3.8 or other versions as well. |
|
I'm confused as to what the products are. So here is the output of |
|
Ok - it seems like python 3.8 and 3.5 didn’t work at all. Python 3.7 and 3.6 partially worked. Your one of the last fix #461 is not included in these wheels. I wonder if that will solve partial failures with 3.6 and 3.7. For 3.8 and 3.5 we have to discuss a bit. Let’s do that next week. |
@ferdonline : could you also upload Dockerfile? |
olupton
pushed a commit
that referenced
this issue
Dec 7, 2022
* Override visit_local_list_statement in CodegenIspcVisitor to declare dt_* as uniform This fixes #329
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When I was trying to upload the wheels to test.pypi.org then I was seeing:
So I thought we have to create
manylinuxwheel. I created following Dockerfile based on manylinux platform:And build this as:
With this we will have wheel:
And then I was able to successfully able to upload to test pypi server:
But when I tried to install this test wheel on BB5 (Red Had 7.4) I see:
So definitely this is not what I thought
manylinuxis about.@ferdonline : I didn't get time to look into this further. You know better and would be great if you could look into this when time permits.
cc: @ohm314
The text was updated successfully, but these errors were encountered: