Fix sod perpendicular velocities

[G-071]

This PR addresses the concerns raised #366.

When all kernels are executed on the same device, the perpendicular velocities are zero. If some kernels are executed on the CPU, others on the GPU there were some values not being zero (though still within machine precision).

This can be avoiding by setting a different fp_contract:

• Changed CMakeList.txt to build the different hydro kernels with the same fp_contract.
• Added CMake Option OCTOTIGER_WITH_FAST_FP_CONTRACT to set the fp_contract to off for all hydro kernels. This also addresses the issue with the LEGACY kernel resulting in non-zero values for sz and zx (even though sy is 0.0).
• Added CMake Option OCTOTIGER_ARCH_FLAG to pick the architecture to compile for. This is a work-around to pick the correct architecture for the files where we are overwriting the cmake compile flags to set a different fp contract.

The issue can also be avoided by always running everything on the GPU:

• Added runtime option DEVICE_ONLY to all host kernels, guaranteeing all kernels of that type to be executed by the GPU.

Additional tests to check for this issue automatically in the future:

• Added additional SOD ctests, checking for sy sz and zx to be zero.
• Added additional SOD test suite running with max_level=3 instead of 1.
• Added Jenkins test, running with fp_contract=off

[diehlpk]

@G-071 I think that @shibersag would be the better choice since he reported the bug.

[diehlpk]

@shibersag and @dmarce1 could you please have a look at this pr?

[shibersag]

@G-071, I made several tests and this PR does not seem to resolve the issue.
I ran on Queen-Bee with:
problem=sod
odt=0.02
stop_time=0.2
gravity=off
max_level=3
#n_species=1
disable_diagnostics=on
xscale=0.5
correct_am_hydro=0
cuda_number_gpus=1
#reflect_bc=1
cuda_streams_per_gpu=32
cuda_buffer_capacity=1
#monopole_host_kernel_type=VC
#multipole_host_kernel_type=VC
#monopole_device_kernel_type=CUDA
#multipole_device_kernel_type=CUDA
hydro_device_kernel_type=CUDA
hydro_host_kernel_type=DEVICE_ONLY

And the output I get is:
sx 4.384558e-03 8.361244e-03
sy 0.000000e+00 0.000000e+00
sz 1.764596e-15 7.976589e-15
zx 4.412263e-16 2.302379e-15

[diehlpk]

@G-071 and @shibersag any update?

[diehlpk]

@shibersag and @dmarce1 Can one of you please review the pull request?

[shibersag]

@diehlpk yeah sure. I had to compile the code again on Queen-Bee and now I will continue with testing this PR.

[shibersag]

I've checked the DEVICE_ONLY option and it indeed does the trick.
Compiling with OCTOTIGER_WITH_FAST_FP_CONTRACT=OFF fixes the issue as well.

See the comments in #366 for more details.
I would recommend adding a test of a long Sod run to detect this kind of issues in the future.

[G-071]

@shibersag Part of the PR already some adds some extended testing with a larger sod problem that uses a larger grid and more time-steps than our usual one. It is being tested multiple times (CPU-only, GPU-only and CPU+GPU). We can make it a bit larger still, but I would not go all out as it is run multiple times for each of our automatic tests (10 build configuration times times all kernel configurations)

[shibersag]

LVGTM!
Thanks @G-071 for working on that.

[shibersag]

I'm merging this PR.
We can discuss about extending the tests in a separate thread.
For now all looks good.