Solver crashes after Elliptic Setup Velocity

[tonyzahtila]

Hi all,

I succesfully run nrspre with,

source /usr/local/module/spartan_new.sh
module load gompic/2019b
module load cmake/3.18.4

#module load OpenMPI/3.1.0-intel-2017.u2-GCC-6.2.0-cuda9-tcp
#module load CMake/3.12.1-intel-2017.u2-GCC-6.2.0-CUDA9

export NEKRS_HOME=$HOME/.local/nekrs
PATH=${NEKRS_HOME}/bin:${PATH}
source $HOME/.bash_profile

#export USE_UCX=0
#export UCX_MEMTYPE_CACHE=n

backend=CUDA

rm -rf ./.cache
nrspre turbPipe 1

I am able to run successfully on 1 GPU. I then request an interactive node and run the following commands,

source /usr/local/module/spartan_new.sh
module load gompic/2019b
module load cmake/3.18.4

backend=CUDA

export NEKRS_HOME=$HOME/.local/nekrs
PATH=${NEKRS_HOME}/bin:${PATH}
source $HOME/.bash_profile

nrsmpi turbPipe 4

The following error message is returned,

================ ELLIPTIC SETUP VELOCITY ================
bID 1 -> bcType fixedValue
bID 2 -> bcType zeroGradient
bID 3 -> bcType zeroValue
allNeumann = 0
[spartan-gpgpu009:21079:0:21079] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x12669a90)
[spartan-gpgpu009:21077:0:21077] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x13473000)
[spartan-gpgpu009:21076:0:21076] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x1257a728)
==== backtrace (tid:  21077) ====
 0 0x0000000000050a75 ucs_debug_print_backtrace()  ???:0
 1 0x00000000002813e0 num_rows()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/sarray_transfer.c:115
 2 0x00000000002813e0 sarray_transfer_many()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/sarray_transfer.c:164
 3 0x0000000000281898 sarray_transfer_()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/sarray_transfer.c:188
 4 0x000000000027d418 shared_ids()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:206
 5 0x000000000027d9c1 get_topology()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:254
 6 0x000000000027d9c1 gs_setup_aux()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:1405
 7 0x000000000027e6a3 gs_setup()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:1442
 8 0x000000000018c96f ogsHostSetup()  /data/cephfs/punim0524/nek/nekRS/3rd_party/gslib/ogs/src/ogsHostSetup.c:61
 9 0x0000000000197d7d ogsSetup()  /data/cephfs/punim0524/nek/nekRS/3rd_party/gslib/ogs/src/ogsSetup.cpp:109
10 0x00000000001174cc ellipticSolveSetup()  /data/cephfs/punim0524/nek/nekRS/src/elliptic/ellipticSolveSetup.cpp:341
11 0x000000000008fc9a nrsSetup()  /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/include/c++/8.3.0/bits/basic_string.tcc:212
12 0x00000000000752a6 nekrs::setup()  ???:0
13 0x000000000040a56d main()  ???:0
14 0x0000000000022555 __libc_start_main()  ???:0
15 0x000000000040aa37 _start()  ???:0
=================================
[spartan-gpgpu009:21077] *** Process received signal ***
[spartan-gpgpu009:21077] Signal: Segmentation fault (11)
[spartan-gpgpu009:21077] Signal code:  (-6)
[spartan-gpgpu009:21077] Failing at address: 0x2c9f00005255
[spartan-gpgpu009:21077] [ 0] /lib64/libpthread.so.0(+0xf630)[0x2b340a811630]
[spartan-gpgpu009:21077] [ 1] /home/tzahtila/.local/nekrs/lib/libnekrs.so(sarray_transfer_many+0x490)[0x2b3409fe03e0]
[spartan-gpgpu009:21077] [ 2] /home/tzahtila/.local/nekrs/lib/libnekrs.so(sarray_transfer_+0x38)[0x2b3409fe0898]
[spartan-gpgpu009:21077] [ 3] /home/tzahtila/.local/nekrs/lib/libnekrs.so(+0x27d418)[0x2b3409fdc418]
[spartan-gpgpu009:21077] [ 4] /home/tzahtila/.local/nekrs/lib/libnekrs.so(+0x27d9c1)[0x2b3409fdc9c1]
[spartan-gpgpu009:21077] [ 5] /home/tzahtila/.local/nekrs/lib/libnekrs.so(gs_setup+0x63)[0x2b3409fdd6a3]
[spartan-gpgpu009:21077] [ 6] /home/tzahtila/.local/nekrs/lib/libnekrs.so(ogsHostSetup+0x7f)[0x2b3409eeb96f]
[spartan-gpgpu009:21077] [ 7] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_Z8ogsSetupiPxRP19ompi_communicator_tiN4occa6deviceE+0x1cd)[0x2b3409ef6d7d]
[spartan-gpgpu009:21077] [ 8] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_Z18ellipticSolveSetupP10elliptic_tN4occa10propertiesE+0x16dc)[0x2b3409e764cc]
[spartan-gpgpu009:21077] [ 9] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_Z8nrsSetupP19ompi_communicator_tN4occa6deviceER9setupAideP5nrs_t+0x9b8a)[0x2b3409deec9a]
[spartan-gpgpu009:21077] [10] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tiiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_S7_+0xcc6)[0x2b3409dd42a6]
[spartan-gpgpu009:21077] [11] /home/tzahtila/.local/nekrs/bin/nekrs[0x40a56d]
[spartan-gpgpu009:21077] [12] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b340aa40555]
[spartan-gpgpu009:21077] [13] /home/tzahtila/.local/nekrs/bin/nekrs[0x40aa37]
[spartan-gpgpu009:21077] *** End of error message ***
==== backtrace (tid:  21076) ====
 0 0x0000000000050a75 ucs_debug_print_backtrace()  ???:0
 1 0x00000000002813e0 num_rows()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/sarray_transfer.c:115
 2 0x00000000002813e0 sarray_transfer_many()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/sarray_transfer.c:164
 3 0x0000000000281898 sarray_transfer_()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/sarray_transfer.c:188
 4 0x000000000027d418 shared_ids()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:206
 5 0x000000000027d9c1 get_topology()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:254
 6 0x000000000027d9c1 gs_setup_aux()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:1405
 7 0x000000000027e6a3 gs_setup()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:1442
 8 0x000000000018c96f ogsHostSetup()  /data/cephfs/punim0524/nek/nekRS/3rd_party/gslib/ogs/src/ogsHostSetup.c:61
 9 0x0000000000197d7d ogsSetup()  /data/cephfs/punim0524/nek/nekRS/3rd_party/gslib/ogs/src/ogsSetup.cpp:109
10 0x00000000001174cc ellipticSolveSetup()  /data/cephfs/punim0524/nek/nekRS/src/elliptic/ellipticSolveSetup.cpp:341
11 0x000000000008fc9a nrsSetup()  /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/include/c++/8.3.0/bits/basic_string.tcc:212
12 0x00000000000752a6 nekrs::setup()  ???:0
13 0x000000000040a56d main()  ???:0
14 0x0000000000022555 __libc_start_main()  ???:0
15 0x000000000040aa37 _start()  ???:0
=================================
[spartan-gpgpu009:21076] *** Process received signal ***
[spartan-gpgpu009:21076] Signal: Segmentation fault (11)
[spartan-gpgpu009:21076] Signal code:  (-6)
[spartan-gpgpu009:21076] Failing at address: 0x2c9f00005254
[spartan-gpgpu009:21076] [ 0] /lib64/libpthread.so.0(+0xf630)[0x2aeb366a6630]
[spartan-gpgpu009:21076] [ 1] /home/tzahtila/.local/nekrs/lib/libnekrs.so(sarray_transfer_many+0x490)[0x2aeb35e753e0]
[spartan-gpgpu009:21076] [ 2] /home/tzahtila/.local/nekrs/lib/libnekrs.so(sarray_transfer_+0x38)[0x2aeb35e75898]
[spartan-gpgpu009:21076] [ 3] /home/tzahtila/.local/nekrs/lib/libnekrs.so(+0x27d418)[0x2aeb35e71418]
[spartan-gpgpu009:21076] [ 4] /home/tzahtila/.local/nekrs/lib/libnekrs.so(+0x27d9c1)[0x2aeb35e719c1]
[spartan-gpgpu009:21076] [ 5] /home/tzahtila/.local/nekrs/lib/libnekrs.so(gs_setup+0x63)[0x2aeb35e726a3]
[spartan-gpgpu009:21076] [ 6] /home/tzahtila/.local/nekrs/lib/libnekrs.so(ogsHostSetup+0x7f)[0x2aeb35d8096f]
[spartan-gpgpu009:21076] [ 7] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_Z8ogsSetupiPxRP19ompi_communicator_tiN4occa6deviceE+0x1cd)[0x2aeb35d8bd7d]
[spartan-gpgpu009:21076] [ 8] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_Z18ellipticSolveSetupP10elliptic_tN4occa10propertiesE+0x16dc)[0x2aeb35d0b4cc]
[spartan-gpgpu009:21076] [ 9] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_Z8nrsSetupP19ompi_communicator_tN4occa6deviceER9setupAideP5nrs_t+0x9b8a)[0x2aeb35c83c9a]
[spartan-gpgpu009:21076] [10] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tiiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_S7_+0xcc6)[0x2aeb35c692a6]
[spartan-gpgpu009:21076] [11] /home/tzahtila/.local/nekrs/bin/nekrs[0x40a56d]
[spartan-gpgpu009:21076] [12] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2aeb368d5555]
[spartan-gpgpu009:21076] [13] /home/tzahtila/.local/nekrs/bin/nekrs[0x40aa37]
[spartan-gpgpu009:21076] *** End of error message ***
==== backtrace (tid:  21079) ====
 0 0x0000000000050a75 ucs_debug_print_backtrace()  ???:0
 1 0x0000000000156628 __memcpy_ssse3_back()  :0
 2 0x0000000000286261 crystal_move()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/crystal.c:90
 3 0x0000000000286261 crystal_router()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/crystal.c:133
 4 0x000000000028164b sarray_transfer_many()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/sarray_transfer.c:161
 5 0x0000000000281898 sarray_transfer_()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/sarray_transfer.c:188
 6 0x000000000027d418 shared_ids()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:206
 7 0x000000000027d9c1 get_topology()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:254
 8 0x000000000027d9c1 gs_setup_aux()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:1405
 9 0x000000000027e6a3 gs_setup()  /data/cephfs/punim0524/nek/nekRS/build/_deps/gs_content-src/src/gs.c:1442
10 0x000000000018c96f ogsHostSetup()  /data/cephfs/punim0524/nek/nekRS/3rd_party/gslib/ogs/src/ogsHostSetup.c:61
11 0x0000000000197d7d ogsSetup()  /data/cephfs/punim0524/nek/nekRS/3rd_party/gslib/ogs/src/ogsSetup.cpp:109
12 0x00000000001174cc ellipticSolveSetup()  /data/cephfs/punim0524/nek/nekRS/src/elliptic/ellipticSolveSetup.cpp:341
13 0x000000000008fc9a nrsSetup()  /usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/include/c++/8.3.0/bits/basic_string.tcc:212
14 0x00000000000752a6 nekrs::setup()  ???:0
15 0x000000000040a56d main()  ???:0
16 0x0000000000022555 __libc_start_main()  ???:0
17 0x000000000040aa37 _start()  ???:0
=================================
[spartan-gpgpu009:21079] *** Process received signal ***
[spartan-gpgpu009:21079] Signal: Segmentation fault (11)
[spartan-gpgpu009:21079] Signal code:  (-6)
[spartan-gpgpu009:21079] Failing at address: 0x2c9f00005257
[spartan-gpgpu009:21079] [ 0] /lib64/libpthread.so.0(+0xf630)[0x2aab3355e630]
[spartan-gpgpu009:21079] [ 1] /lib64/libc.so.6(+0x156628)[0x2aab338c1628]
[spartan-gpgpu009:21079] [ 2] /home/tzahtila/.local/nekrs/lib/libnekrs.so(crystal_router+0x2c1)[0x2aab32d32261]
[spartan-gpgpu009:21079] [ 3] /home/tzahtila/.local/nekrs/lib/libnekrs.so(sarray_transfer_many+0x6fb)[0x2aab32d2d64b]
[spartan-gpgpu009:21079] [ 4] /home/tzahtila/.local/nekrs/lib/libnekrs.so(sarray_transfer_+0x38)[0x2aab32d2d898]
[spartan-gpgpu009:21079] [ 5] /home/tzahtila/.local/nekrs/lib/libnekrs.so(+0x27d418)[0x2aab32d29418]
[spartan-gpgpu009:21079] [ 6] /home/tzahtila/.local/nekrs/lib/libnekrs.so(+0x27d9c1)[0x2aab32d299c1]
[spartan-gpgpu009:21079] [ 7] /home/tzahtila/.local/nekrs/lib/libnekrs.so(gs_setup+0x63)[0x2aab32d2a6a3]
[spartan-gpgpu009:21079] [ 8] /home/tzahtila/.local/nekrs/lib/libnekrs.so(ogsHostSetup+0x7f)[0x2aab32c3896f]
[spartan-gpgpu009:21079] [ 9] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_Z8ogsSetupiPxRP19ompi_communicator_tiN4occa6deviceE+0x1cd)[0x2aab32c43d7d]
[spartan-gpgpu009:21079] [10] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_Z18ellipticSolveSetupP10elliptic_tN4occa10propertiesE+0x16dc)[0x2aab32bc34cc]
[spartan-gpgpu009:21079] [11] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_Z8nrsSetupP19ompi_communicator_tN4occa6deviceER9setupAideP5nrs_t+0x9b8a)[0x2aab32b3bc9a]
[spartan-gpgpu009:21079] [12] /home/tzahtila/.local/nekrs/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tiiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_S7_+0xcc6)[0x2aab32b212a6]
[spartan-gpgpu009:21079] [13] /home/tzahtila/.local/nekrs/bin/nekrs[0x40a56d]
[spartan-gpgpu009:21079] [14] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2aab3378d555]
[spartan-gpgpu009:21079] [15] /home/tzahtila/.local/nekrs/bin/nekrs[0x40aa37]
[spartan-gpgpu009:21079] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 21077 on node spartan-gpgpu009 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Any help appreciated.

[tonyzahtila]

Hey, @stgeke , you can close this, I (am new to nekRS) and I was not aware that I'd have to change the .udf for a periodic pipe. Works now. Thanks for reaching out to Sean to offer assistance.