vowpalwabbit: fix build against boost-python. #45987

danieldk · 2018-09-03T13:51:18Z

Patch setup.py to look for libboost_python{Major}{Minor}. This fixes the build in unstable and 18.09.

Motivation for this change

Going through the list of #45960 I found that the Vowpal Wabbit Python module does not compile. This fixes compilation with Python 2.7 and 3.6.

Note: for some reason, boost is added to the build environment without boost-python, unless I explicitly add python.pkgs.boost as a dependency.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Patch setup.py to look for libboost_python{Major}{Minor}.

vcunat · 2018-09-03T14:20:45Z

@GrahamcOfBorg build python2Packages.vowpalwabbit python3Packages.vowpalwabbit

vcunat · 2018-09-03T14:20:51Z

/cc maintainer @teh.

FRidh · 2018-09-03T17:22:18Z

Note: for some reason, boost is added to the build environment without boost-python, unless I explicitly add python.pkgs.boost as a dependency.

That's indeed what should be used.

FRidh · 2018-09-03T17:22:59Z

@GrahamcOfBorg build python2Packages.vowpalwabbit python3Packages.vowpalwabbit

GrahamcOfBorg · 2018-09-03T18:26:13Z

Failure on x86_64-darwin (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

    self._build_extensions_serial()
  File "/nix/store/dlswl7xjvh3bsfydygqzi2n0f82p5d9p-python3-3.6.6/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "setup.py", line 114, in build_extension
    env['PYTHON_LIBS'] = '-l {}'.format(find_boost())
  File "setup.py", line 39, in find_boost
    raise Exception('Could not find boost python library')
Exception: Could not find boost python library
builder for '/nix/store/vfvjr3lfs9c10gp5yddlzhfmx4fr7gjw-python3.6-vowpalwabbit-8.5.0.drv' failed with exit code 1
error: build of '/nix/store/fq9xcrp3x7lwlg30j73cfq5yalwcj88i-python2.7-vowpalwabbit-8.5.0.drv', '/nix/store/vfvjr3lfs9c10gp5yddlzhfmx4fr7gjw-python3.6-vowpalwabbit-8.5.0.drv' failed

GrahamcOfBorg · 2018-09-03T18:26:13Z

Failure on aarch64-linux (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

    self._build_extensions_serial()
  File "/nix/store/sb88r9crdhinhkm6vpxwzf5zmij86p8v-python3-3.6.6/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "setup.py", line 115, in build_extension
    subprocess.check_call(['make', 'python'], cwd=path.join(here, 'src'), env=env)
  File "/nix/store/sb88r9crdhinhkm6vpxwzf5zmij86p8v-python3-3.6.6/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', 'python']' returned non-zero exit status 2.
builder for '/nix/store/yyryl97m3xcpzk5a524rf0jk4qsgvbn1-python3.6-vowpalwabbit-8.5.0.drv' failed with exit code 1
error: build of '/nix/store/9p5k6dq76kvpdzgcjbzr35lmw99wrk94-python2.7-vowpalwabbit-8.5.0.drv', '/nix/store/yyryl97m3xcpzk5a524rf0jk4qsgvbn1-python3.6-vowpalwabbit-8.5.0.drv' failed

GrahamcOfBorg · 2018-09-03T18:26:15Z

Success on x86_64-linux (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

shrinking RPATHs of ELF executables and libraries in /nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0
shrinking /nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0/lib/python3.6/site-packages/pylibvw.cpython-36m-x86_64-linux-gnu.so
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0/lib
patching script interpreter paths in /nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0
checking for references to /build in /nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0...
running install tests
running check
/nix/store/n6qm2ygsvb25nr7rmsq6rpkmhs6cvxfp-python2.7-vowpalwabbit-8.5.0
/nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0

- Follow the same logic for finding boost-python as for Linux. - Set DYLD_LIBRARY_PATH so that ctypes find_library can find boost-python. - Do not use '-mt' suffix when linking against boost-serialization and boost-program_options on Darwin.

danieldk · 2018-09-03T19:51:48Z

The added commit should also fix the build on Darwin (tested on macOS 10.14 beta).

xeji · 2018-09-03T21:27:10Z

@GrahamcOfBorg build python2Packages.vowpalwabbit python3Packages.vowpalwabbit

GrahamcOfBorg · 2018-09-03T21:27:55Z

Failure on aarch64-linux (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

    self._build_extensions_serial()
  File "/nix/store/sb88r9crdhinhkm6vpxwzf5zmij86p8v-python3-3.6.6/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "setup.py", line 113, in build_extension
    subprocess.check_call(['make', 'python'], cwd=path.join(here, 'src'), env=env)
  File "/nix/store/sb88r9crdhinhkm6vpxwzf5zmij86p8v-python3-3.6.6/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', 'python']' returned non-zero exit status 2.
builder for '/nix/store/68w132wx8hfhpiq0b3rfqrfz5z02wik9-python3.6-vowpalwabbit-8.5.0.drv' failed with exit code 1
error: build of '/nix/store/68w132wx8hfhpiq0b3rfqrfz5z02wik9-python3.6-vowpalwabbit-8.5.0.drv', '/nix/store/827xwn1q8ygmvy0pa2gya2x13kvqr0lp-python2.7-vowpalwabbit-8.5.0.drv' failed

xeji · 2018-09-03T21:30:53Z

This never built on aarch64 according to hydra logs, so would you please mark it as broken on aarch64 to get rid of one more failure?

GrahamcOfBorg · 2018-09-03T21:30:58Z

Success on x86_64-linux (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

shrinking RPATHs of ELF executables and libraries in /nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0
shrinking /nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0/lib/python3.6/site-packages/pylibvw.cpython-36m-x86_64-linux-gnu.so
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0/lib
patching script interpreter paths in /nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0
checking for references to /build in /nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0...
running install tests
running check
/nix/store/jy6d5va2gbm7dhj42lvyx09pjf3h1a3y-python2.7-vowpalwabbit-8.5.0
/nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0

GrahamcOfBorg · 2018-09-03T21:34:38Z

Success on x86_64-darwin (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

Successfully installed vowpalwabbit-8.5.0
~
post-installation fixup
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/jp3cdh73crchd25d2m26d3g75yhmp0bz-python3.6-vowpalwabbit-8.5.0/lib
patching script interpreter paths in /nix/store/jp3cdh73crchd25d2m26d3g75yhmp0bz-python3.6-vowpalwabbit-8.5.0
running install tests
running check
/nix/store/94i1pvdzkbl5f8jyrik01m01n3s8jh0d-python2.7-vowpalwabbit-8.5.0
/nix/store/jp3cdh73crchd25d2m26d3g75yhmp0bz-python3.6-vowpalwabbit-8.5.0

danieldk · 2018-09-04T05:44:31Z

Ok, marked it broken on aarch64.

xeji · 2018-09-04T07:01:30Z

backported in 46f4a93..b447979

vowpalwabbit: fix build against boost-python.

875bb6b

Patch setup.py to look for libboost_python{Major}{Minor}.

danieldk requested a review from FRidh as a code owner September 3, 2018 13:51

GrahamcOfBorg added 6.topic: python 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Sep 3, 2018

vowpal-wabbit: fix Darwin build.

fe7e89b

- Follow the same logic for finding boost-python as for Linux. - Set DYLD_LIBRARY_PATH so that ctypes find_library can find boost-python. - Do not use '-mt' suffix when linking against boost-serialization and boost-program_options on Darwin.

vowpalwabbit: mark broken on aarch64.

69d418a

xeji merged commit d22e22e into NixOS:master Sep 4, 2018

danieldk deleted the fix-vowpalwabbit branch September 5, 2018 14:25

danieldk restored the fix-vowpalwabbit branch September 5, 2018 14:25

danieldk deleted the fix-vowpalwabbit branch September 5, 2018 14:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vowpalwabbit: fix build against boost-python. #45987

vowpalwabbit: fix build against boost-python. #45987

danieldk commented Sep 3, 2018 •

edited

Loading

vcunat commented Sep 3, 2018

vcunat commented Sep 3, 2018

FRidh commented Sep 3, 2018

FRidh commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

danieldk commented Sep 3, 2018

xeji commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

xeji commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

danieldk commented Sep 4, 2018

xeji commented Sep 4, 2018

vowpalwabbit: fix build against boost-python. #45987

vowpalwabbit: fix build against boost-python. #45987

Conversation

danieldk commented Sep 3, 2018 • edited Loading

Motivation for this change

Things done

vcunat commented Sep 3, 2018

vcunat commented Sep 3, 2018

FRidh commented Sep 3, 2018

FRidh commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

danieldk commented Sep 3, 2018

xeji commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

xeji commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

GrahamcOfBorg commented Sep 3, 2018

danieldk commented Sep 4, 2018

xeji commented Sep 4, 2018

danieldk commented Sep 3, 2018 •

edited

Loading