Non-deterministic wrong result from tensordot

[manopapad]

This program (derived from tests/tensordot.py):

import legate.numpy as lg
import numpy as np

a = lg.random.rand(3, 5, 4).astype(np.float16)
b = lg.random.rand(4, 5, 3).astype(np.float16)

a = lg.random.rand(3, 5, 4).astype(np.float16)
b = lg.random.rand(5, 4, 3).astype(np.float16)
cn = np.tensordot(a, b)
print('cn', flush=True)
print(cn, flush=True)
c = lg.tensordot(a, b)
print('c', flush=True)
print(c, flush=True)

assert np.allclose(cn, c)

when run as follows:

LEGATE_TEST=1 legate 79.py -lg:numpy:test --cpus 4

fails about 20% of the time, with:

cn
[[4.07  4.83  5.01 ]
 [4.2   4.562 5.863]
 [4.344 4.52  3.914]]
c
[[4.07  4.83  5.01 ]
 [4.2   4.562 5.863]
 [4.344 4.52  3.916]]
[0 - 700005133000]    0.946367 {6}{python}: python exception occurred within task:
Traceback (most recent call last):
  File "/Users/mpapadakis/legate.core/install/lib/python3.8/site-packages/legion_top.py", line 410, in legion_python_main
    run_path(args[start], run_name='__main__')
  File "/Users/mpapadakis/legate.core/install/lib/python3.8/site-packages/legion_top.py", line 234, in run_path
    exec(code, module.__dict__, module.__dict__)
  File "79.py", line 16, in <module>
    assert np.allclose(cn, c)
AssertionError

[magnatelee]

I feel discrepancies like this with float16 are inevitable, because of region reductions being non-deterministic. Do we know that this issue reproduces on a single CPU? Otherwise, I wouldn't worry too much about this issue.

[lightsighter]

I think @manopapad and I have convinced ourselves that this is a precision issue. np.allclose has a fixed tolerance regardless of the underlying types. The resulting value under an error run is 1 ulp away from the "correct" value. It's highly likely then that something got rounded off depending on the ordering. The solution should be to relax the tolerance for np.allclose for 16-bit floating point types in accordance with their reduced precision.

[manopapad]

@marcinz It sounds like 0.1% relative accuracy is a more reasonable expectation when dealing with 16-bit floating point types. Could you change np.allclose on test runs using float16, so we don't get spurious failures in CI?

[manopapad]

I have ported @marcinz's fix from branch-21.10