stripping some manylinux generated files produces broken shared objects

[gotyaoi]

Attempting to strip some of the .so files from some manylinux generated packages will run without error messages, but the resulting file is broken. ldd claims it is "not a dynamic executable" and trying to import it from python gives "ELF load command address/offset not properly aligned".

perhaps related to #78

Here's an example with scipy/special/_ufuncs.so, though the problem affects about 21 of the 91 .so files in scipy

$ cat /etc/os-release
NAME="Amazon Linux AMI"
VERSION="2017.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2017.03"
PRETTY_NAME="Amazon Linux AMI 2017.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2017.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
$ uname -a
Linux ip-10-40-23-202 4.9.27-14.31.amzn1.x86_64 #1 SMP Wed May 10 01:58:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ strip --version
GNU strip version 2.23.52.0.1-55.65.amzn1 20130226
Copyright 2013 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.

steps to reproduce:

$ virtualenv reproducer
New python executable in reproducer/bin/python2.7
Also creating executable in reproducer/bin/python
Installing setuptools, pip...done.
$ . reproducer/bin/activate
(reproducer)$ pip install -U pip setuptools
You are using pip version 6.0.8, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip from https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
  Using cached pip-9.0.1-py2.py3-none-any.whl
Collecting setuptools from https://pypi.python.org/packages/fc/cb/748dcabb152fbc7eceaf585d052b8bc11cf028b917a7d2343cf1e0c092c3/setuptools-36.0.1-py2.py3-none-any.whl#md5=7a52500dcfd7c4f37f5d20e462c93560
  Using cached setuptools-36.0.1-py2.py3-none-any.whl
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 12.0.5
    Uninstalling setuptools-12.0.5:
      Successfully uninstalled setuptools-12.0.5

  Found existing installation: pip 6.0.8
    Uninstalling pip-6.0.8:
      Successfully uninstalled pip-6.0.8

Successfully installed pip-9.0.1 setuptools-36.0.1
(reproducer)$ pip install scipy
Collecting scipy
  Using cached scipy-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting numpy>=1.8.2 (from scipy)
  Using cached numpy-1.13.0-cp27-cp27mu-manylinux1_x86_64.whl
Installing collected packages: numpy, scipy
Successfully installed numpy-1.13.0 scipy-0.19.1
(reproducer)$ ldd reproducer/lib64/python2.7/site-packages/scipy/special/_ufuncs.so
	linux-vdso.so.1 =>  (0x00007ffdbd1e2000)
	libopenblasp-r0-39a31c03.2.18.so => /home/ec2-user/reproducer/lib64/python2.7/site-packages/scipy/special/../.libs/libopenblasp-r0-39a31c03.2.18.so (0x00007f447ea8f000)
	libgfortran-ed201abd.so.3.0.0 => /home/ec2-user/reproducer/lib64/python2.7/site-packages/scipy/special/../.libs/libgfortran-ed201abd.so.3.0.0 (0x00007f447e794000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f447e48a000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f447e274000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f447deb1000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f447dc95000)
	/lib64/ld-linux-x86-64.so.2 (0x000055b50199b000)
(reproducer)$ python
Python 2.7.12 (default, Sep  1 2016, 22:14:00)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy.special import logit
>>> ^D
(reproducer)$ strip reproducer/lib64/python2.7/site-packages/scipy/special/_ufuncs.so
(reproducer)$ ldd reproducer/lib64/python2.7/site-packages/scipy/special/_ufuncs.so
	not a dynamic executable
(reproducer)$ python
Python 2.7.12 (default, Sep  1 2016, 22:14:00)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy.special import logit
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ec2-user/reproducer/local/lib64/python2.7/site-packages/scipy/special/__init__.py", line 640, in <module>
    from ._ufuncs import *
ImportError: /home/ec2-user/reproducer/local/lib64/python2.7/site-packages/scipy/special/_ufuncs.so: ELF load command address/offset not properly aligned
>>> ^D
(reproducer)$ ^D

[njsmith]

This must be a bug in patchelf. NixOS/patchelf#117 is the PR that allegedly reworked how it works to make binaries strippable...

We might also want to reconsider having auditwheel strip binaries, to sidestep the whole mess around stripping-after-patchelf.

[njsmith]

CC @rpurdie as the author of that patchelf PR, in case they have some idea.

[ogrisel]

We might also want to reconsider having auditwheel strip binaries, to sidestep the whole mess around stripping-after-patchelf.

I am not opposed to that pragmatic choice.

[rmcgibbo]

We might also want to reconsider having auditwheel strip binaries, to sidestep the whole mess around stripping-after-patchelf.

Stripping by default in auditwheel sounds like a bad idea, but if patchelf works properly after stripping, I would be fine with adding stripping as an option to auditwheel.

[gotyaoi]

Still occuring in scipy as of 1.0.0. Comparing to a package that doesn't have the issue, like numpy, it appears that scipy is using a significantly older https://github.com/matthew-brett/multibuild, https://github.com/matthew-brett/multibuild/tree/e6ebbfa42588c26579eff99c50e84326dd9f6b3e in scipy as compared to https://github.com/matthew-brett/multibuild/tree/07480be3e0b3490495cb8a9629e55be54c3adac3 in numpy, so maybe that could have something to do with it? At this point I'm not even sure at what point in the build chain this issue might be getting introduced, but getting any traction on this would be a big help to us.

[njsmith]

I still think it makes sense for auditwheel to strip by default (though of course with an option to disable that). One option would be to write such a patch for auditwheel. Alternatively if Robert is still opposed, a patch to allow opt-in stripping plus patches to the scipy build scripts to enable that option would probably be accepted. I don't think anyone else is likely to address this soon if you don't. On Nov 7, 2017 18:28, "gotyaoi" <[email protected]> wrote: Still occuring in scipy as of 1.0.0. Comparing to a package that doesn't have the issue, like numpy, it appears that scipy is using a significantly older https://github.com/matthew-brett/multibuild, https://github.com/matthew-brett/multibuild/tree/ e6ebbfa42588c26579eff99c50e84326dd9f6b3e in scipy as compared to https://github.com/matthew-brett/multibuild/tree/ 07480be3e0b3490495cb8a9629e55be54c3adac3 in numpy, so maybe that could have something to do with it? At this point I'm not even sure at what point in the build chain this issue might be getting introduced, but getting any traction on this would be a big help to us. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#119 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAlOaNIBu1BEwvjismctVbSiu9AXxEqcks5s0PW4gaJpZM4OF7T7> .

[tkelman]

getting any traction on this would be a big help to us

Likewise. Where's the best place to find (and reproduce) how scipy's manylinux release wheels are built? For anyone else who's hitting this a crude workaround for now is avoiding stripping the problematic binaries via something to the effect of

find /path/to/installed/libs -name "*.so" | grep -v ufuncs | grep -v fblas | grep -v flapack | \
        grep -v cython_blas | grep -v cython_lapack | grep -v ellip_harm | \
        grep -v odepack | grep -v quadpack | grep -v vode | grep -v lsoda | \
        grep -v iterative | grep -v superlu | grep -v arpack | grep -v trlib | \
        grep -v lbfgs | grep -v qhull | xargs strip

but that doesn't help with size as much as I'd like. If stripping before running patchelf will work, I may create my own wheels with that performed. It's not easy to find where the pypi released wheels are built from though.

[matthew-brett]

Scipy's wheels built from https://github.com/MacPython/scipy-wheels

[tkelman]

Thanks @matthew-brett, I'm trying to dissect that enough to re-run a build myself either locally or on CI so I can manually add a strip call before auditwheel/patchelf corrupt things. I tried with the latest nightly build at http://8b8c47f30575e674b56d-47bd50c35cd79bd838daf386af554a83.r59.cf2.rackcdn.com/scipy-1.1.0.dev0+20171221163716_bdc562f-cp27-cp27mu-manylinux1_x86_64.whl and it also has this issue, despite being built with a more recent multibuild as far as I can tell from MacPython/scipy-wheels@db2510b

[tkelman]

https://github.com/tkelman/scipy-wheels/raw/v1.0.0-stripped-before-auditwheel/wheelhouse/scipy-1.0.0-cp27-cp27mu-manylinux1_x86_64.whl seems to work so far, generated via tkelman/multibuild@dd1220b and tkelman/scipy-wheels@ce9385e. I hope that looks sane to people who are familiar with the wheel-building process. It's roughly half the size of the non-stripped release wheels.

One thing to note is that you'll have to exclude even these already-stripped scipy binaries from any later attempts to strip them, since that would break again.

[tkelman]

I'll note that libhdf5-b2d0cb23.so.10.2.1 from the manylinux wheel for h5py 2.7.1 has the same problem if you try to strip *.so* instead of just *.so.

[dazza-codes]

Similar to @tkelman - using a work-around to exclude some files from the strip, but hope to track progress on this issue for an upstream solution. While trying to optimize lambda layer build for python3.6 on lambci, using something like this:

strip_binary_libs() {
  site=$1
  echo "Optimizing binary libraries *.so.* in $site ..."
  find "$site" -type f \( iname '*.so.*' ! iname '*libgfortran*' ! -iname '*libgdal*' ! -iname '*libproj*' \) -exec strip {} \;
  echo "Optimizing cpython compiled libraries in $site ..."
  find "$site" -type f \( iname '*.cpython*.so' ! iname '*netCDF4*' \) -exec strip {} \;
}

package_dst=$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')
strip_binary_libs "$package_dst"

Running a project test suite on the venv indicates whether or not the venv will work for the project involved. Some packages in the venv may have various levels of optimization for lazy-loading modules that are not touched by the project - those modules might load stripped libraries that are broken, but never imported by the project. It's not a comprehensive solution, only a trial-n-error exclusion for the project concerned (rather than a test of all packages in the venv).

[AaronDMarasco]

Running into this as I'm trying to package an RPM with PySide in it on CentOS 7... I've been working around these patchelf bugs for years now. 😢

[mayeut]

The option --strip (opt-in) has been added in auditwheel 3.3.0.
This shall fix this issue.