terminate called after throwing an instance of 'occa::exception'

[okcfdlab]

Hello, I'm trying to run nekRS with 1 GPU on UBC Sockeye HPC. I configured and built NekRS without incident. When running in an interactive GPU job, nekrs --setup turbPipe throws the following error:

terminate called after throwing an instance of 'occa::exception'
  what():
---[ Error ]--------------------------------------------------------------------
    File     : /home/jbrinker/nekRS-21.1/3rd_party/occa/src/occa/internal/modes/cuda/device.cpp
    Line     : 347
    Function : compileKernel
    Message  : Error compiling [myDummyKernelName], Command: [nvcc -O3 --fmad=true -arch=sm_70 -ptx -I/home/jbrinker/.local/nekrs/include -I/home/jbrinker/.local/nekrs/include -L/home/jbrinker/.local/nekrs/lib -locca -x cu /scratch/pr-jbrinker-1/jbrinker/nekRS/turbPipe/.cache/occa/cache/ddbb9775537d70da/source.cpp -o /scratch/pr-jbrinker-1/jbrinker/nekRS/turbPipe/.cache/occa/cache/ddbb9775537d70da/binary > /dev/null 2>&1]
    Stack
      12 occa::error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      11 occa::cuda::device::compileKernel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, occa::json const&, occa::io::lock_t&)
      10 occa::cuda::device::buildKernelFromProcessedSource(occa::hash_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, occa::lang::sourceMetadata_t&, occa::lang::sourceMetadata_t&, occa::json const&, occa::io::lock_t)
       9 occa::launchedModeDevice_t::buildKernel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, occa::hash_t, bool, occa::json const&)
       8 occa::launchedModeDevice_t::buildKernel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, occa::hash_t, occa::json const&)
       7 occa::device::buildKernel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, occa::json const&) const
       6 occa::device::buildKernelFromString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, occa::json const&) const
       5 compileKernels()
       4 nekrs::setup(ompi_communicator_t*, int, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
       3 nekrs()
       2 /usr/lib64/libc.so.6(__libc_start_main+0xf5)
       1 nekrs()
================================================================================

I cannot figure out how to debug or increase the verbosity to troubleshoot this error. Can anyone provide a suggestion?

[tcew]

You can see the nvcc compiler errors directly by running the compilation command shown in the output:

nvcc -O3 --fmad=true -arch=sm_70 -ptx -I/home/jbrinker/.local/nekrs/include -I/home/jbrinker/.local/nekrs/include -L/home/jbrinker/.local/nekrs/lib -locca -x cu /scratch/pr-jbrinker-1/jbrinker/nekRS/turbPipe/.cache/occa/cache/ddbb9775537d70da/source.cpp -o /scratch/pr-jbrinker-1/jbrinker/nekRS/turbPipe/.cache/occa/cache/ddbb9775537d70da/binary

[okcfdlab]

Thanks Tim. Indeed, that uncovered the issue: I had compiled NekRS with gcc/9.1.0 and GCC versions greater than 7 are not supported. Further work is needed on my end to configure the software environment, it seems.

Does the requirement of 6.2<= GCC <= v7 make sense for others?

[stgeke]

This is not a nekRS specific restriction. The supported gcc version depends on the CUDA version. See e.g. here https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html.

[okcfdlab]

Ah, I see. Thanks @stgeke. I'll try again with a more recent CUDA version.

[okcfdlab]

Thanks @tcew and @stgeke for your help. I think I have NekRS built correctly now with CUDA 11.3 and gcc/9.4.0. When I try to execute nekrs --setup turbPipe I get the following error message:

Scanning dependencies of target UDF
In file included from /arc/software/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-9.1.0-mj7s6dgfnhgi2n42fyxgmitnuslcyol3/include/c++/9.1.0/bits/std_abs.h:38,
                 from /arc/software/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-9.1.0-mj7s6dgfnhgi2n42fyxgmitnuslcyol3/include/c++/9.1.0/cmath:47,
                 from /arc/software/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-9.1.0-mj7s6dgfnhgi2n42fyxgmitnuslcyol3/include/c++/9.1.0/math.h:36,
                 from /scratch/pr-jbrinker-1/jbrinker/nekRS/turbPipe/.cache/udf/udf.cpp:4:
/usr/include/stdlib.h:32:10: fatal error: stddef.h: No such file or directory
   32 | #include <stddef.h>
      |          ^~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/UDF.dir/udf.cpp.o] Error 1
make[1]: *** [CMakeFiles/UDF.dir/all] Error 2
make: *** [all] Error 2

My ignorant googling of the error message suggests a version mismatch between gcc-core and gcc. Does this look like a further configuration problem?

[tcew]

That's a new one. Not got much for this one.

Did you remove all objects and libraries in the code stack and rebuild after changing NVIDIA and gcc versions ?

[okcfdlab]

I believe so. I deleted the build directory, then reconfigured and rebuilt.

[okcfdlab]

I've pinged the cluster admin as well, in case this is a software configuration issue.

[okcfdlab]

I figured it out. In turbPipe.udf, I needed to replace #include <math.h> with #include "math.h". Otherwise, it kept loading the default system gcc version and not the module that I had loaded. With that change the turbPipe job is running great, currently on one V100 GPU.

Thanks @tcew and @stgeke for your help.