You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
oneDNN validation for Nvidia backend hits a correctness issue with reorder under benchdnn.
Version
Latest master.
Environment
Hardware:
NVIDIA A100 80GB PCIe
(A10 should also work for most cases).
Software
SYCL Compiler with Nvidia support.
Any version that compiles without issues, preferable no later than April.
[Optional] TBB
Any version.
[Optional] OpenCL CPU
Latest version is preferable.
Optional means that CPU backend can be enabled if dependency is satisfied. Otherwise, should be switched off.
CUDA_LOGINFO_DBG=1 CUDA_LOGDEST_DBG=stdout -- enables cuda API dump
CUDNN_LOGINFO_DBG=1 CUDNN_LOGDEST_DBG=stdout -- enables cudnn API dump
DNNL_VERBOSE=all (or desired level) -- enables oneDNN execution information
Helper tips:
benchdnn supports verbosity through -vX. Most info is available at v6. It's possible to dump destination with -v99 when really needed.
benchdnn documentation is here: https://github.com/oneapi-src/oneDNN/tree/master/tests/benchdnn (scroll down). Reorder doc and others may be found through links.
benchdnn binary also supports --help command, which will tip to use --reorder --help to dump all supported options.
Observed behavior
Failures may not be reproduced by a single run, running batch is the only reliable way to hit the issue (at least so far).
This is the most impactful issue so far because it affects many other primitives which are reorder-based.
Preliminary analysis says that it's likely a cross engine reorder implementation that affects the final result. The main suspect is synchronization part for out-of-order queue (since it's a default queue for Nvidia backend).
In-order queue seems to work fine (at this point, can be enabled inside the library manually only, benchdnn will get an extension with option soon).
dzarukin
added
bug
A confirmed library bug
and removed
sighting
Suspicious library behavior. Should be promoted to a bug when confirmed
labels
Aug 14, 2023
Since this commit, the failures will no longer be observed, but the out-of-order workflow is still not properly working. Will submit another issue to focus on out-of-order.
Summary
oneDNN validation for Nvidia backend hits a correctness issue with reorder under benchdnn.
Version
Latest master.
Environment
Hardware:
NVIDIA A100 80GB PCIe
(A10 should also work for most cases).
Software
SYCL Compiler with Nvidia support.
Any version that compiles without issues, preferable no later than April.
[Optional] TBB
Any version.
[Optional] OpenCL CPU
Latest version is preferable.
Steps to reproduce
Build
mkdir -p build
cd build
cmake .. -DCMAKE_BUILD_TYPE=release (or debug) -DDNNL_CPU_RUNTIME=DPCPP (or NONE) -DDNNL_GPU_RUNTIME=DPCPP -DDNNL_GPU_VENDOR=NVIDIA -DONEDNN_BUILD_GRAPH=OFF
cmake --build . --target benchdnn
Run
<env_vars> ./build/tests/benchdnn/benchdnn --reorder --engine=gpu --batch=test_reorder_gpu
Helper env vars:
CUDA_LOGINFO_DBG=1 CUDA_LOGDEST_DBG=stdout -- enables cuda API dump
CUDNN_LOGINFO_DBG=1 CUDNN_LOGDEST_DBG=stdout -- enables cudnn API dump
DNNL_VERBOSE=all (or desired level) -- enables oneDNN execution information
Helper tips:
benchdnn supports verbosity through -vX. Most info is available at v6. It's possible to dump destination with -v99 when really needed.
benchdnn documentation is here: https://github.com/oneapi-src/oneDNN/tree/master/tests/benchdnn (scroll down). Reorder doc and others may be found through links.
benchdnn binary also supports
--help
command, which will tip to use--reorder --help
to dump all supported options.Observed behavior
Failures may not be reproduced by a single run, running batch is the only reliable way to hit the issue (at least so far).
This is the most impactful issue so far because it affects many other primitives which are reorder-based.
Preliminary analysis says that it's likely a cross engine reorder implementation that affects the final result. The main suspect is synchronization part for out-of-order queue (since it's a default queue for Nvidia backend).
In-order queue seems to work fine (at this point, can be enabled inside the library manually only, benchdnn will get an extension with option soon).
Most of failures look like this:
Expected behavior
The issue is not appearing during batch validation.
The text was updated successfully, but these errors were encountered: