Make runtest failure #2320

deepsamal · 2015-04-17T01:09:48Z

Mac OS 10.9, CUDA 7.0
Make all and Make test ran without errors, except some warning about unused -pthread and 1 warning in test. But $make runtest gives segmentation fault. $make sudo runtest gives the error-
.build_release/tools/caffe
dyld: Library not loaded: @rpath/libcudart.7.0.dylib
Referenced from: /Users/deepsamal/Desktop/Research/caffe/.build_release/tools/caffe
Reason: image not found
make: *** [runtest] Trace/BPT trap: 5

Seems a problem with dynamic loading of the cuda library path.
checking the links of libcaffe.so, the paths to cuda libs are not resolved. But the libraries are present both in /usr/local/cuda/lib and in /Developer/NVIDIA.... folders.

Deep-Samals-MacBook-Pro:caffe deepsamal$ otool -L ~/Desktop/Research/caffe/build/lib/libcaffe.so
/Users/deepsamal/Desktop/Research/caffe/build/lib/libcaffe.so:
@rpath/libcaffe.so (compatibility version 0.0.0, current version 0.0.0)
@rpath/libcudart.7.0.dylib (compatibility version 0.0.0, current version 7.0.29)
@rpath/libcublas.7.0.dylib (compatibility version 0.0.0, current version 7.0.29)
@rpath/libcurand.7.0.dylib (compatibility version 0.0.0, current version 7.0.29)
/usr/local/lib/libglog.0.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/local/lib/libprotobuf.9.dylib (compatibility version 10.0.0, current version 10.1.0)
/usr/local/lib/libleveldb.1.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/lib/libsnappy.1.dylib (compatibility version 4.0.0, current version 4.0.0)
/usr/local/lib/liblmdb.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/lib/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
libhdf5_hl.9.dylib (compatibility version 10.0.0, current version 10.0.0)
libhdf5.9.dylib (compatibility version 10.0.0, current version 10.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
/usr/local/opt/opencv/lib/libopencv_core.2.4.dylib (compatibility version 2.4.0, current version 2.4.11)
/usr/local/opt/opencv/lib/libopencv_highgui.2.4.dylib (compatibility version 2.4.0, current version 2.4.11)
/usr/local/opt/opencv/lib/libopencv_imgproc.2.4.dylib (compatibility version 2.4.0, current version 2.4.11)
/usr/local/lib/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libcudnn.6.5.dylib (compatibility version 0.0.0, current version 6.5.48)
/usr/local/lib/libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)
libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

Also I looked at links of the libcudart library-

$ otool -L /usr/local/cuda/lib/libcudart.7.0.dylib
/usr/local/cuda/lib/libcudart.7.0.dylib:
@rpath/libcudart.7.0.dylib (compatibility version 0.0.0, current version 7.0.29)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.17.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

In both the outputs, library links seem to be broken. But make all and test are fine, this means dynamic linking is okay, loading is not?

May be the problem is environment variable related, my variables are-
$ echo $DYLD_FALLBACK_LIBRARY_PATH
/usr/local/cuda/lib:/Developer/NVIDIA/CUDA-7.0/lib:/Users/deepsamal/anaconda/lib:/usr/local/lib:/usr/lib:/opt/intel/composer_xe_2015.2.132/mkl/lib/:/opt/intel/composer_xe_2015.2.132/compiler/lib

$ echo $DYLD_LIBRARY_PATH
/usr/local/cuda/lib

$ echo $LD_LIBRARY_PATH
/usr/local/cuda/lib:/opt/intel/composer_xe_2015.2.132/mkl/lib/

$ ECHO $PATH
/Developer/NVIDIA/CUDA-7.0/bin:/Users/deepsamal/anaconda/bin:/Applications/MATLAB_R2014a.app/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin

All advised paths are included, in fact, I even tried without setting LD_LIBRARY_PATH, same error. Any idea what might be the cause?

iwen-kang · 2015-04-18T01:46:45Z

I have exactly the same problem on my MacBookPro OS 10.10, CUDA 7.0. Any help will be much appreciated, thanks!
=========== terminal output ===========
sudo make runtest
.build_release/tools/caffe
dyld: Library not loaded: @rpath/libcudart.7.0.dylib
Referenced from: /Developer/caffe/.build_release/tools/caffe
Reason: image not found
make: *** [runtest] Trace/BPT trap: 5

output from "env | grep LIB":
DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib:/Applications/anaconda/lib:/usr/local/lib:/usr/lib
LD_LIBRARY_PATH=/usr/local/lib
DYLD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-7.0/cudnn-6.5-osx-v2:/Developer/NVIDIA/CUDA-7.0/lib

shelhamer · 2015-04-27T07:33:43Z

Follow the installation instructions and ask installation questions on the caffe-users group. You should not set DYLD_LIBRARY_PATH.

MartinHjelm · 2015-06-02T12:25:23Z

This is bit old I guess. And I shouldn't be spamming with install advice but I just had the same problem. At the caffe user forum https://groups.google.com/forum/#!msg/caffe-users/BkMspfbYLIE/Pi6SUVANAKUJ someone said fix the DYLD_FALLBACK_LIBRARY_PATH variable for me it was just to add the /usr/local/cuda/lib as the first path

export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/lib:/usr/lib:/Developer/NVIDIA/CUDA-7.0/lib:

This worked for me.

wlike · 2015-08-01T04:07:50Z

set DYLD_FALLBACK_LIBRARY_PATH solved my problem too

gsabran · 2015-10-13T05:58:34Z

I've set my DYLD_FALLBACK_LIBRARY_PATH as suggested and followed the advices from robertsdionne:
echo $DYLD_FALLBACK_LIBRARY_PATH
/usr/local/cuda/lib:/usr/local/lib:/Users/guigou/caffe/distribute/lib:/Developer/NVIDIA/CUDA-7.5/lib:

and I still get

ImportError                               Traceback (most recent call last)
<ipython-input-1-1cca3aa1f8c5> in <module>()
----> 1 import caffe

/Users/guigou/caffe/python/caffe/__init__.py in <module>()
----> 1 from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
      2 from ._caffe import set_mode_cpu, set_mode_gpu, set_device, Layer, get_solver, layer_type_list
      3 from .proto.caffe_pb2 import TRAIN, TEST
      4 from .classifier import Classifier
      5 from .detector import Detector

/Users/guigou/caffe/python/caffe/pycaffe.py in <module>()
     11 import numpy as np
     12 
---> 13 from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
     14         RMSPropSolver, AdaDeltaSolver, AdamSolver
     15 import caffe.io

ImportError: dlopen(/Users/guigou/caffe/python/caffe/_caffe.so, 2): Library not loaded: @rpath/libcudart.7.5.dylib
  Referenced from: /Users/guigou/caffe/python/caffe/_caffe.so
  Reason: image not found

qiuwch · 2015-10-18T00:08:22Z

I have the exactly the same issue. I also already set

DYLD_LIBRARY_PATH
DYLD_FALLBACK_LIBRARY_PATH

My version is also CUDA7.5. I think when I used CUDA7.0 before, everything is fine.

dg2 · 2015-10-20T16:45:37Z

Caffe seems to be linking in a pretty funny way ... If I run

otool -L .build_release/tools/caffe

I see something like this

.build_release/tools/caffe:
    @rpath/libcaffe.so (compatibility version 0.0.0, current version 0.0.0)
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
    @rpath/libcudart.7.5.dylib (compatibility version 0.0.0, current version 7.5.20)
    @rpath/libcublas.7.5.dylib (compatibility version 0.0.0, current version 7.5.20)
    @rpath/libcurand.7.5.dylib (compatibility version 0.0.0, current version 7.5.20)
    /usr/local/opt/glog/lib/libglog.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/opt/gflags/lib/libgflags.2.dylib (compatibility version 2.0.0, current version 2.1.2)
    /usr/local/lib/libprotobuf.9.dylib (compatibility version 10.0.0, current version 10.1.0)
    /usr/local/lib/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.1.1)
    libhdf5_hl.10.dylib (compatibility version 11.0.0, current version 11.1.0)
    libhdf5.10.dylib (compatibility version 11.0.0, current version 11.1.0)
        ...

So some library paths are static, others are run-time paths and others are just assumed to be in the executable path (which they aren't). Unfortunately, in my case at least there are not rpath's set for either the executables or the libcaffe :( Just check `otool -l .build_release/tools/caffe | grep RPATH'

If you need a quick and dirty fix, you can add your CUDA path as an rpath

install_name_tool -add_rpath /usr/local/cuda/lib .build_release/tools/caffe
install_name_tool -add_rpath /usr/local/cuda/lib .build_release/test/test_all.testbin
install_name_tool -add_rpath /usr/local/cuda/lib .build_release/lib/libcaffe.so

With that I can run make runtest, but you should do the same for a few more files (mostly every caffe binary and also libcaffe.so) if you want to use them. And I also expect trouble with the HDF5 libraries at some point, but you can also use install_name_tool to fix those issues. Anyway, the most reasonable option is to fix the Makefile to do a proper job in OS X.

cecilpang · 2015-10-29T04:09:05Z

If you are installing on OS X El Capitan, DYLD_FALLBACK_LIBRARY_PATH is cleared by the new System Integrity Protection feature of El Capitan:

https://developer.apple.com/library/watchos/documentation/Security/Conceptual/System_Integrity_Protection_Guide/System_Integrity_Protection_Guide.pdf

make runtest completed successfully after I disabled System Integrity Protection.

a7744hsc · 2015-10-30T12:07:47Z

@cecilpang Thanks for your tips and it really helps. here is the steps to disable Integrity Protection:

Boot to Recovery OS by restarting your machine and holding down the Command and R keys at startup.
Launch Terminal from the Utilities menu.
Enter the following command:
$ csrutil disable
4.restart done

jiajunshen · 2015-12-16T08:46:33Z

@a7744hsc @cecilpang Your solutions resolved my problem. Thanks!

a7744hsc · 2015-12-17T10:14:27Z

@jiajunshen Very glad to hear that

cchriste · 2015-12-18T00:16:02Z

Worked... but so annoying! And now what about my system's integrity? Will I ever be able to trust it again?

shal233 · 2016-03-02T00:51:54Z

Before you run 'make runtest', do 'make clean' once.

fanshiqing · 2016-05-18T14:00:09Z

@a7744hsc @cecilpang Your solutions resolved my problem too. Many thanks!
BTW, Is there any other possible walk-around without disabling the Integrity Protection of Mac OS?

kmhofmann · 2016-06-04T11:36:18Z

Shouldn't this maybe be reopened, since there is no good fix for this?
(I don't consider disabling the Mac OS X Integrity Protection a good fix...)

mkmohangb · 2016-06-18T21:04:37Z

Please refer #3227 for a workaround without disabling Integrity protection on OSX 10.11.

apuljain · 2016-11-21T08:33:00Z

What should be the fix if there is no cuda gpu?

shelhamer closed this as completed Apr 27, 2015

lettergram mentioned this issue Feb 11, 2016

OSX 10.11 linking issue: dyld: Library not loaded: libcaffe.so.1.0.0-rc3 and dyld: Library not loaded: @rpath/libcudart.7.5.dylib #3628

Closed

nnop mentioned this issue Dec 5, 2016

DYLD_xxx and SIP nnop/notes#44

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make runtest failure #2320

Make runtest failure #2320

deepsamal commented Apr 17, 2015

iwen-kang commented Apr 18, 2015

shelhamer commented Apr 27, 2015

MartinHjelm commented Jun 2, 2015

wlike commented Aug 1, 2015

gsabran commented Oct 13, 2015

qiuwch commented Oct 18, 2015

dg2 commented Oct 20, 2015

cecilpang commented Oct 29, 2015

a7744hsc commented Oct 30, 2015

jiajunshen commented Dec 16, 2015

a7744hsc commented Dec 17, 2015

cchriste commented Dec 18, 2015

shal233 commented Mar 2, 2016

fanshiqing commented May 18, 2016

kmhofmann commented Jun 4, 2016

mkmohangb commented Jun 18, 2016

apuljain commented Nov 21, 2016

Make runtest failure #2320

Make runtest failure #2320

Comments

deepsamal commented Apr 17, 2015

iwen-kang commented Apr 18, 2015

shelhamer commented Apr 27, 2015

MartinHjelm commented Jun 2, 2015

wlike commented Aug 1, 2015

gsabran commented Oct 13, 2015

qiuwch commented Oct 18, 2015

dg2 commented Oct 20, 2015

cecilpang commented Oct 29, 2015

a7744hsc commented Oct 30, 2015

jiajunshen commented Dec 16, 2015

a7744hsc commented Dec 17, 2015

cchriste commented Dec 18, 2015

shal233 commented Mar 2, 2016

fanshiqing commented May 18, 2016

kmhofmann commented Jun 4, 2016

mkmohangb commented Jun 18, 2016

apuljain commented Nov 21, 2016