2023-09-17 21:17:40.944285: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED

[af-74413592]

ubuntu18.04 python 3.7 tensorflow-gpu 1.15.2 cuda 10.0 gpu RTX 4090

failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
Traceback (most recent call last):
File "/root/miniconda3/envs/adnerf/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/root/miniconda3/envs/adnerf/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/root/miniconda3/envs/adnerf/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(15998, 494), b.shape=(494, 2048), m=15998, n=2048, k=494
[[{{node deepspeech/MatMul}}]]
[[deepspeech/logits/_25]]
(1) Internal: Blas GEMM launch failed : a.shape=(15998, 494), b.shape=(494, 2048), m=15998, n=2048, k=494
[[{{node deepspeech/MatMul}}]]
0 successful operations.
0 derived errors ignored.