Build and Run on OLCF Frontier

Build without MPI enabled using hipcc

Setup

Create a top level directory and run all of the following commands from it (unless otherwise noted).

Environment

Create a file named envHipccFrontier.sh with the following contents:

module load amd-mixed/6.2.0
export CRAYPE_LINK_TYPE=dynamic #kokkos configure warns about this
unset HIP_PATH

Source the file each time you start a new session:

source envHipccFrontier.sh

Build Kokkos

Create a file named installKokkos.sh with the following contents:

git clone -b develop [email protected]:Kokkos/kokkos.git # tested with 6d2005a5c

buildType=$1
[[ $buildType != "release" && $buildType != "debug" && $buildType != "RelWithDebInfo" ]] && 
  echo " buildType != [release|debug|RelWithDebInfo] " &&
  exit 1

compiler=$2
[[ $compiler != "amdclang" && $compiler != "hipcc" ]] && 
  echo " compiler != [amdclang|hipcc] " &&
  exit 1

cxx=""
[[ $compiler == "amdclang" ]] && cxx="CC"
[[ $compiler == "hipcc" ]] && cxx="hipcc"

bdir=$PWD/buildKokkosVega90a_${buildType}_${compiler}
 
cmake -S ./kokkos -B $bdir \
 -DCMAKE_BUILD_TYPE=${buildType}\
 -DCMAKE_CXX_COMPILER=${cxx} \
 -DCMAKE_CXX_STANDARD=17 \
 -DCMAKE_CXX_EXTENSIONS=OFF\
 -DKokkos_ENABLE_TESTS=OFF\
 -DKokkos_ENABLE_EXAMPLES=OFF\
 -DKokkos_ENABLE_SERIAL=ON\
 -DKokkos_ENABLE_OPENMP=OFF\
 -DKokkos_ENABLE_HIP=ON\
 -DKokkos_ARCH_VEGA90A=ON\
 -DKokkos_ENABLE_DEBUG=OFF\
 -DCMAKE_INSTALL_PREFIX=$bdir/install

cmake --build $bdir -j8 --target install

Make it executable:

chmod +x installKokkos.sh 

Run it:

./installKokkos.sh release hipcc

Build LibMeshb -- Optional --

git clone [email protected]:LoicMarechal/libMeshb.git # tested with b4a9151

buildType=release
bdir=$PWD/buildLibMeshbVega90a_${buildType}

cmake -S ./libMeshb -B $bdir \
 -DCMAKE_BUILD_TYPE=${buildType}\
 -DCMAKE_C_COMPILER=hipcc \
 -DCMAKE_INSTALL_PREFIX=$bdir/install

cmake --build $bdir --target install -j8

Build Omega_h

The following was tested with reducedThrust@2745b6b4.

Create a file named buildOmegah.sh with the following contents:

git clone -b reducedThrust [email protected]:SCOREC/omega_h.git

buildType=$1
[[ $buildType != "release" && $buildType != "debug" && $buildType != "RelWithDebInfo" ]] && 
  echo " buildType != [release|debug|RelWithDebInfo] " &&
  exit 1

mpi=off

compiler=$2
[[ $compiler != "amdclang" && $compiler != "hipcc" ]] && 
  echo " compiler != [amdclang|hipcc] " &&
  exit 1

libmesh=$3
[[ $libmesh != "on" && $libmesh != "off" ]] && 
  echo " libmesh != [on|off] " &&
  exit 1

libmeshArgs=""
[[ $libmesh == "on" ]] && libmeshArgs="-DlibMeshb_PREFIX=$PWD/buildLibMeshbVega90a_release/install"

cxx=""
[[ $compiler == "amdclang" ]] && cxx="CC"
[[ $compiler == "hipcc" ]] && cxx="hipcc"

bdir=$PWD/buildOmegahVega90a_MPI${mpi}_${buildType}_${compiler}
 
cmake -S ./omega_h -B $bdir \
-DCMAKE_BUILD_TYPE=${buildType} \
-DCMAKE_INSTALL_PREFIX=$bdir/install \
-DBUILD_SHARED_LIBS=OFF \
-DOmega_h_USE_MPI=$mpi \
-DOmega_h_USE_OpenMP=OFF \
-DCMAKE_CXX_COMPILER=${cxx} \
-DOmega_h_USE_Kokkos=ON \
-DKokkos_PREFIX=$PWD/buildKokkosVega90a_${buildType}_${compiler}/install \
-DOmega_h_USE_libMeshb=${libmesh} \
${libmeshArgs} \
-DBUILD_TESTING=ON

cmake --build $bdir -j8

Make it executable:

chmod +x buildOmegah.sh

Run it:

./buildOmegah.sh release hipcc off #last argument disables libmeshb

Test

cd buildOmegahVega90a_* #edit this
srun -A CSC262 -t 00:02:00 -q debug -N 1 -n 1 -c1 --gpus-per-task=1 ctest #this will block until the allocation is available

All tests should pass.

Build with MPI enabled using Cray compiler wrappers and AMD compilers

Setup

Create a top level directory and run all of the following commands from it (unless otherwise noted).

Environment

Create a file named envFrontierAmd.sh with the following contents:

module load PrgEnv-amd

Source the file each time you start a new session:

source envFrontierAmd.sh

Build Kokkos

Create a file named installKokkos.sh with the following contents:

git clone -b 4.0.01 [email protected]:Kokkos/kokkos.git

bdir=$PWD/buildKokkosVega90a_amd
 
cmake -S kokkos -B $bdir \
 -DCMAKE_BUILD_TYPE=RelWithDebInfo\
 -DCMAKE_CXX_COMPILER=CC\
 -DCMAKE_CXX_EXTENSIONS=OFF\
 -DKokkos_ENABLE_TESTS=OFF\
 -DKokkos_ENABLE_EXAMPLES=OFF\
 -DKokkos_ENABLE_SERIAL=ON\
 -DKokkos_ENABLE_OPENMP=OFF\
 -DKokkos_ENABLE_HIP=ON\
 -DKokkos_ARCH_VEGA90A=ON\
 -DKokkos_ENABLE_DEBUG=OFF\
 -DCMAKE_INSTALL_PREFIX=$bdir/install

cmake --build $bdir -j8 --target install

Make it executable:

chmod +x installKokkos.sh

Run it:

./installKokkos.sh

Build Omega_h

The following was tested with reducedThrust@5e333078.

Create a file named installOmegah.sh with the following contents:

git clone -b reducedThrust [email protected]:SCOREC/omega_h.git

bdir=$PWD/buildOmegahVega90a_amd
 
cmake -S omega_h -B $bdir \
-DCMAKE_INSTALL_PREFIX=$bdir/install \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DBUILD_SHARED_LIBS=OFF \
-DOmega_h_USE_CUDA=OFF \
-DOmega_h_USE_MPI=ON \
-DOmega_h_USE_OpenMP=OFF \
-DCMAKE_CXX_COMPILER=CC \
-DOmega_h_USE_Kokkos=ON \
-DKokkos_PREFIX=$PWD/buildKokkosVega90a_amd/install \
-DBUILD_TESTING=ON

cmake --build $bdir -j8

Make it executable:

chmod +x installOmegah.sh

Run it:

./installOmegah.sh

Test

cd buildOmegahVega90a_amd
salloc -A CSC262 -t 00:10:00 -q debug -N 1 #this will block until the allocation is available
#once the allocation is granted
ctest -E warp #exclude warp_test_serial (hangs) and warp_test_parallel (fails)

All tests that are executed should pass.