cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED

[lorentz-wu]

my cuda is 10
when i run script/smalst_op_run.sh , i get some error

outputs = self.forward(in_data)
File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward
y = _matmul(a, b, self.transa, self.transb)
File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul
return xp.matmul(a, b)
File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul
File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul
File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched
File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched
File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status
cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED

how can i solve it?

[silviazuffi]

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10.

…

On 14. Feb 2020, at 03:36, lorentz-wu ***@***.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ>.

[lorentz-wu]

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10.
…
On 14. Feb 2020, at 03:36, lorentz-wu @.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ.

thank you for your help

[lorentz-wu]

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10.
…
On 14. Feb 2020, at 03:36, lorentz-wu @.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ.

dear silviazuffi,
could you tell me your cuda's version? I will try the cuda with same version

[silviazuffi]

Yes, 9.2

…

On 18. Feb 2020, at 02:14, lorentz-wu ***@***.***> wrote: The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10. … <x-msg://14/#> On 14. Feb 2020, at 03:36, lorentz-wu @.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6 <#6>?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ <https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ>. dear silviazuffi, could you tell me your cuda's version? I will try the cuda with same version — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABNYVFOXSAXGVOIZMW7CL4TRDMZALA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMAGLBQ#issuecomment-587228550>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNYVFL2CYSPGYL3VWRXNELRDMZALANCNFSM4KU7KARQ>.

[QingAL]

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10.
…
On 14. Feb 2020, at 03:36, lorentz-wu @.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ.

dear silviazuffi,
could you tell me your cuda's version? I will try the cuda with same version

hi,Did you solve this problem when use cuda9.2?
my cuda version is 9.0 also get this error