[Bug]: Tau build failure from E4S Testsuite on E4S 22.11

[shahzebsiddiqui]

CDASH Build

https://my.cdash.org/test/81709519

Link to buildspec file

https://github.com/buildtesters/buildtest-nersc/blob/devel/buildspecs/e4s/E4S-Testsuite/perlmutter/22.11/tau.yml

Please describe the issue?

@wspear this tau test is failing

Relevant log output

Debug: Linking with TAU Options
Executing> /opt/cray/pe/gcc/11.2.0/bin/gfortran -o ../bin/lu.W.4 lu.o init_comm.o read_input.o bcast_inputs.o proc_grid.o neighbors.o nodedim.o subdomain.o setcoeff.o sethyper.o setbv.o exact.o setiv.o erhs.o ssor.o exchange_1.o exchange_3.o exchange_4.o exchange_5.o exchange_6.o rhs.o l2norm.o jacld.o blts.o jacu.o buts.o error.o pintgr.o verify.o ../common/print_results.o ../common/timers.o -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-fqug27dng5axhcwkzpv2lb2vtadxutan/craycnl/lib/shared-gnu-papi-mpi-pthread-python-pdt -lTAU -lTAU-pthread -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-fqug27dng5axhcwkzpv2lb2vtadxutan/craycnl/lib/shared-gnu-papi-mpi-pthread-python-pdt -L/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -lfmpich -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-fqug27dng5axhcwkzpv2lb2vtadxutan/craycnl/lib -lTauMpi-gnu-papi-mpi-pthread-python-pdt -lmpi -Wl,-rpath,/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/lib -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/lib -lpapi -lpthread -ldl -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.38-eylvvrez76ep2uimlooqhpd32s5czuxe/lib -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.38-eylvvrez76ep2uimlooqhpd32s5czuxe/lib64 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.38-eylvvrez76ep2uimlooqhpd32s5czuxe/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.38-eylvvrez76ep2uimlooqhpd32s5czuxe/lib64 -lbfd -liberty -lz -ldl -lrt -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libunwind-1.6.2-67uzawqr5ucusdwuqckuz56stdn7nz5l/lib -lunwind -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libunwind-1.6.2-67uzawqr5ucusdwuqckuz56stdn7nz5l/lib -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.8.15-7gnip4cbtzsukrx7krgd5gsc6njsghop/lib -lpython3.8 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.8.15-7gnip4cbtzsukrx7krgd5gsc6njsghop/lib -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/lib -lotf2 -lotf2 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/lib -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/elfutils-0.187-p4nc4kahegufpr3jspa7h247z2jb6t3l/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/elfutils-0.187-p4nc4kahegufpr3jspa7h247z2jb6t3l/lib -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-ed2wwuwmtgakhctjpkhgbatsetpbzxo2/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-ed2wwuwmtgakhctjpkhgbatsetpbzxo2/lib -ldwarf -lz -lelf -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-fqug27dng5axhcwkzpv2lb2vtadxutan/craycnl/lib -lTauMpi-gnu-papi-mpi-pthread-python-pdt -L/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -lmpi -Wl,-rpath,/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -L/lib -lstdc++ -lgcc_eh -L/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-fqug27dng5axhcwkzpv2lb2vtadxutan/craycnl/lib/static-gnu-papi-mpi-pthread-python-pdt -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-fqug27dng5axhcwkzpv2lb2vtadxutan/craycnl/lib/shared-gnu-papi-mpi-pthread-python-pdt -ldl

make[1]: Leaving directory '/global/cfs/cdirs/m3503/buildtest/runs/perlmutter_scheduled_test/2023-06-01/perlmutter.slurm.regular/tau/tau_e4s_testsuite_22.11/1442a3a5/stage/testsuite/validation_tests/tau/LU'
+ ls ./bin/lu.W.4
./bin/lu.W.4
Running /global/cfs/cdirs/m3503/buildtest/runs/perlmutter_scheduled_test/2023-06-01/perlmutter.slurm.regular/tau/tau_e4s_testsuite_22.11/1442a3a5/stage/testsuite/validation_tests/tau
Skipping load: Environment already setup
MPICH ERROR [Rank 0] [job id 9831750.0] [Thu Jun  1 19:54:56 2023] [nid002345] - Abort(-1) (rank 0 in comm 0): MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked
 (Other MPI error)

aborting job:
MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked


Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
srun: error: nid002345: task 0: Exited with exit code 255
srun: Terminating StepId=9831750.0
slurmstepd: error: *** STEP 9831750.0 ON nid002345 CANCELLED AT 2023-06-02T02:54:56 ***
srun: error: nid002345: tasks 1-7: Terminated
srun: Force Terminated StepId=9831750.0
Run failed