Under OS X 10.11 El Capitan, when Caffe is called from scripts it cannot find its libraries.

[orsharir]

(Originally written as comment in #2720)
It appears that El Capitan made changes to how the OS treat DYLD_ prefix variables - specifically, it seems that the OS unset the variables when running scripts (e.g. the ipython command is usually a script, or the 'make runtests' command). I couldn't find any official documentation about this behavior, but there are references from other open source projects:

• https://github.com/kubo/fix_oralib_osx
• http://www.wine-reviews.net/2015/10/winebottler-1752-has-been-released-now.html

Without treating the above issue, when Caffe is ran from a script, it's unable to find some of its libraries (e.g. libcudart.7.5.dylib) and then quites.

It seems that the long term solution is to remove the dependency on the DYLD_FALLBACK_LIBRARY_PATH environment variable, as suggested by the second link above:

First, Apple got rid of LD_LIBRARY_PATH and DYLD_FALLBACK_LIBRARY_PATH. Forcing us to “otool -L” and “install_name_tool -change” thru all the bins and libs of Wine to use proper @ rpath relative paths for all Wine related libraries. The result is, of course, a correct linking through out the app – thank you Apple. I guess somebody has to kick you once a while to come up with proper solutions.

For the near term you can turn off System Integrity Protection (see a guide here), which return the system to the previous behavior, but leave your system vulnerable to all kind of attacks. You can also try to work around some of the issues by creating .bashenv script (similar to .bash_profile) where you set your DYLD_FALLBACK_LIBRARY_PATH variable, and add to your .bash_profile the line 'export BASH_ENV=~/.bashenv' - this forces the system to run .bashenv before every non-interactive bash session (i.e. bash scripts). This last part resolved the problems I had with ipython, but still doesn't resolve the issue of running 'make runtest'.

[nathanntg]

Unfortunately, I am not proficient enough with make files and compiler parameters to achieve a fix for this, but I was able to find a work around that may also help someone else solve this issue. I wan't to avoid disabling System Integrity Protection, so I instead looked at using the binary tools to relink the caffe library and command line executable to the CUDA libraries.

This assumes CUDA 7.5 is installed in "/usr/local/cuda" and that make && make test have already been run. From the main directory where you ran make, the following scripts will relink the binary and the library:

cd build/tools
otool -L caffe | grep rpath
cp caffe caffe_backup # backup
install_name_tool -change "@rpath/libcudart.7.0.dylib" "/usr/local/cuda/lib/libcudart.7.5.dylib" caffe
install_name_tool -change "@rpath/libcublas.7.0.dylib" "/usr/local/cuda/lib/libcublas.7.5.dylib" caffe
install_name_tool -change "@rpath/libcurand.7.5.dylib" "/usr/local/cuda/lib/libcurand.7.5.dylib" caffe

cd ../lib
otool -L libcaffe.so | grep rpath
cp libcaffe.so libcaffe_backup.so # backup
install_name_tool -change "@rpath/libcudart.7.0.dylib" "/usr/local/cuda/lib/libcudart.7.5.dylib" libcaffe.so
install_name_tool -change "@rpath/libcublas.7.0.dylib" "/usr/local/cuda/lib/libcublas.7.5.dylib" libcaffe.so
install_name_tool -change "@rpath/libcurand.7.5.dylib" "/usr/local/cuda/lib/libcurand.7.5.dylib" libcaffe.so

cd ../test
cp test_all.testbin test_all_backup.testbin # backup
install_name_tool -change "@rpath/libcudart.7.0.dylib" "/usr/local/cuda/lib/libcudart.7.5.dylib" test_all.testbin
install_name_tool -change "@rpath/libcublas.7.0.dylib" "/usr/local/cuda/lib/libcublas.7.5.dylib" test_all.testbin
install_name_tool -change "@rpath/libcurand.7.5.dylib" "/usr/local/cuda/lib/libcurand.7.5.dylib" test_all.testbin

I hope that helps.

[djsutherland]

Here's a version of @nathanntg's script that should handle every binary built by caffe, and also fixes libcudnn if you're using it:

https://gist.github.com/dougalsutherland/886dd23202ebcf21d2b8

[recastrodiaz]

On OSX El Capitan, I added one line to the Makefile after the exisiting LDFLAGS. No need to run @dougalsutherland's script:

LDFLAGS += $(foreach librarydir,$(LIBRARY_DIRS),-L$(librarydir)) $(PKG_CONFIG) \
                $(foreach library,$(LIBRARIES),-l$(library))
LDFLAGS += -Wl,-rpath,/usr/local/cuda/lib

Source: Forges0 @developers.nvidia.com

I can make a PR, but I'm not sure how this change would affect other OS'es?

[shelhamer]

Thanks for noting this @orsharir. Fixed in #3695, #3696.

[diegorep]

@shelhamer I had been tracking down a similar bug with the hdf5 libraries and at last found a solution here. It turns out that 'make runtest' also breaks down when using anaconda python (and the hdf5 libraries included in it) in OS X El Capitan. @nathanntg 's manual linking solution worked for me. It may be worth setting a separate HDF5_LIB/DIR variable in the Makefile's config.

For those who have the same problem I did: I had a number of linking problems when trying to use anaconda python with brew-installed hdf5, and somehow my pycaffe imports segfaulted when using both system and homebrew pythons and their respective hdf5s. The solution was to use anaconda python only and doing the manual link described above, this time changing both "@rpath/./libhdf5.10.dylib" to "$HOME/anaconda/lib/libhdf5.10.dylib" and "@rpath/./libhdf5_hl.10.dylib" to "$HOME/anaconda/lib/libhdf5_hl.10.dylib"

[dozyc]

Thanks @diegorep, your post and @nathanntg's helped me fix my problems with libhdf5(_hl) missing.

[zhangboyu]

Thanks @diegorep , your post about the hdf5 really helps me out! Thank you so much!

[llcdefgab]

Thanks @recastrodiaz , your post solves my problem!