Provide performance portable (multi-architecture) Kokkos implementation for compressible hydrodynamics (second order, Godunov, MUSCL-Hancock) on cartesian grids.
- Kokkos library will be built by euler_kokkos using the same flags (architecture, optimization, ...).
- cmake with version >= 3.X (3.X is chosen to meet Kokkos own requirement for CMake; i.e. it might increase in the future)
Current application is configured with kokkos library as a git submodule. So you'll need to run the following git commands right after cloning euler_kokkos:
git submodule init
git submodule updateKokkos is built with the same flags as the main application.
A few example builds, with minimal configuration options.
Just make sure that your env variable CMAKE_PREFIX_PATH point to the location where Kokkos where installed. More precisely if Kokkos is installed in KOKKOS_ROOT, you add $KOKKOS_ROOT/lib/cmake to your CMAKE_PREFIX_PATH; this way kokkos will be found automagically by cmake, and the right Kokkos backend will be selected.
mkdir -p build; cd build
cmake -DEULER_KOKKOS_BUILD_KOKKOS=OFF ..
make- Create a build directory, configure and make
mkdir build; cd build
cmake -DEULER_KOKKOS_USE_MPI=OFF -DEULER_KOKKOS_BUILD_KOKKOS=ON -DEULER_KOKKOS_BACKEND=OpenMP ..
make -j 4Add variable CXX on the cmake command line to change the compiler (clang++, icpc, pgcc, ....).
- Create a build directory, configure and make
mkdir build; cd build
# If you are compiling and running on the same host, you can omit architecture flags,
# Kokkos will detect the GPU architecture available on your paltform
cmake -DEULER_KOKKOS_USE_MPI=OFF -DEULER_KOKKOS_BUILD_KOKKOS=ON -DEULER_KOKKOS_BACKEND=Cuda -DKokkos_ARCH_MAXWELL50=ON ..
make -j 4nvcc_wrapper is a compiler wrapper arroud NVIDIA nvcc. It is available from Kokkos sources: external/kokkos/bin/nvcc_wrapper. Any Kokkos application target NVIDIA GPUs must be built with nvcc_wrapper.
Please make sure to use a CUDA-aware MPI implementation (OpenMPI or MVAPICH2) built with the proper flags for activating CUDA support.
It may happen that eventhough your MPI implementation is actually cuda-aware, cmake find_package macro for MPI does not detect it to be cuda aware. In that case, you can enforce cuda awareness by turning option EULER_KOKKOS_USE_MPI_CUDA_AWARE_ENFORCED to ON.
You don't need to use mpi compiler wrapper mpicxx, cmake should be able to correctly populate MPI_CXX_INCLUDE_PATH, MPI_CXX_LIBRARIES which are passed to all final targets.
- Create a build directory, configure and make
mkdir build; cd build
cmake -DEULER_KOKKOS_USE_MPI=ON -DEULER_KOKKOS_BUILD_KOKKOS=ON -DEULER_KOKKOS_BACKEND=Cuda -DKokkos_ARCH_MAXWELL50=ON ..
make -j 4Example command line to run the application (1 GPU used per MPI task)
mpirun -np 4 ./euler_kokkos ./test_implode_2D_mpi.iniMake sure to have rocm/hip tools with version at least 5.2 if build againt kokkos 4.0.
For example:
mkdir build/hip; cd build/hip
export CXX=hipcc
cmake -DEULER_KOKKOS_USE_MPI=ON -DEULER_KOKKOS_BUILD_KOKKOS=ON -DEULER_KOKKOS_BACKEND=HIP -DKokkos_ARCH_VEGA90A=ON ..
make -j 4Make sure to have CMake variable CMAKE_EXPORT_COMPILE_COMMANDS set to ON, it will generate a file named compile_commands.json.
Then you can symlink the generated file in the top level source directory.
Please visit :
A Sphinx/html documentation will (hopefully) soon be populated.
To build it:
mkdir -p build/doc
cd build/doc
# build doxygen documentation
cmake .. -DEULER_KOKKOS_BUILD_DOC:BOOL=ON -DEULER_KOKKOS_DOC_TYPE:STRING=doxygen
# build sphinx/html documentation
cmake .. -DEULER_KOKKOS_BUILD_DOC:BOOL=ON -DEULER_KOKKOS_DOC_TYPE:STRING=htmlBuilding sphinx documentation requires to have python3 with up-to-date breathe extension.