Could not build with CUDA_ARCH_LIST=Common

[hubutui]

Description

I could not build with CUDA_ARCH_LIST=Common on ArchLinux.

Error Message

mxnet-git.log

To Reproduce

  cmake_opts=(
    -DBUILD_CPP_EXAMPLES:BOOL=OFF
    -DBUILD_TESTING:BOOL=OFF
    -DCMAKE_BUILD_TYPE:String=Release
    -DCMAKE_EXE_LINKER_FLAGS="$(pkg-config --libs cblas)"
    -DCMAKE_INSTALL_LIBDIR:PATH=lib
    -DCMAKE_INSTALL_PREFIX:PATH=/usr
    -DCMAKE_SHARED_LINKER_FLAGS="$(pkg-config --libs cblas)"
    -DCMAKE_SKIP_INSTALL_RPATH:BOOL=ON
    -DINSTALL_DOCUMENTATION:BOOL=OFF
    -DUSE_BLAS=open
    -DUSE_CPP_PACKAGE:BOOL=ON
    -DUSE_DIST_KVSTORE:BOOL=ON
    -DUSE_GPERFTOOLS:BOOL=OFF
    -DUSE_JEMALLOC:BOOL=OFF
    -DUSE_LAPACK:BOOL=ON
    -DUSE_MKL_IF_AVAILABLE:BOOL=OFF
    -DUSE_OPENMP:BOOL=ON
    -DUSE_S3:BOOL=ON
)
  cmake \
    ${cmake_opts[@]} \
    -DCUDA_ARCH_LIST=${cudaarch} \
    -DUSE_CUDA:BOOL=ON \
    -DUSE_CUDNN:BOOL=ON \
    -DUSE_MKLDNN:BOOL=ON \
    -DUSE_NCCL:BOOL=ON \
    -DUSE_OPENCV:BOOL=OFF \
    ..
  make
  cd ../python
  python setup.py build --with-cython

Environment info

Same as #15263 (comment)

What have you tried to solve it?

1. It's OK to build with CUDA_ARCH_LIST=Pascal
2. I try CI fails with PC-relative offset overflow in PLT entry #15971 (comment), and add -mcmodel=large to both CMAKE_C_FLAGS and CMAKE_CXX_FLAGS, but with no luck.
3. MXNet Master build for CUDA with DEBUG=1 failing #14263 (comment) mentioned a solution, but I'm not sure how to do with cmake building.

[samskalicky]

@mxnet-label-bot update [Question, Build]

[hubutui]

I also check #15981, but it seems it's not fixed yet. According to https://github.com/apache/incubator-mxnet/blob/master/CMakeLists.txt#L51, ENABLE_TESTCOVERAGE=OFF by default.

[leezu]

Try building binutils locally with --enable-64-bit-archive.
#14263 (comment)

[hubutui]

@leezu Hey, I try rebuilt binutils locally with --enable-64-bit-archive, and then build mxnet. It didn't work. I could confirm that CMAKE_AR=/usr/bin/ar, and this is the newly build binutils.

[leezu]

@hubutui besides using the 64 bit ar with CMAKE_AR and -mcmodel=medium, you can also add -Wl,--no-relax. In my case, the error message instructed to specify -Wl,--no-relax.

Also, for cmake, I find that using cmake -GNinja to select ninja build system makes the relocation truncation issue less likely

[hubutui]

You mean I could try with -mcmodel=large -Wl --no-relax or -mcmodel=medium -Wl --no-relax again?

[leezu]

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mcmodel=medium -Wl,--no-relax")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mcmodel=medium -Wl,--no-relax")

the , is required

[hubutui]

@leezu You mean just append these two line to CMakeLists.txt? I have try it, but with no lock. I'm currently manual set CUDA_COMMON_GPU_ARCHITECTURES to compute capability >= 3.7.