Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

richardson_lucy example crashes with large filter size #276

Closed
eddy16112 opened this issue Apr 13, 2022 · 4 comments
Closed

richardson_lucy example crashes with large filter size #276

eddy16112 opened this issue Apr 13, 2022 · 4 comments
Assignees

Comments

@eddy16112
Copy link

The example is https://github.com/nv-legate/cunumeric/blob/branch-22.05/examples/richardson_lucy.py
If running with richardson_lucy.py -t -fx=512 -fy=256 -fz=151 -x=2048 -y=1024 -z=604, then I am running to the following errors:

Internal cuFFT failure with error code 4 in file cunumeric/convolution/convolve.cu at line 1133
Internal cuFFT failure with error code 1 in file cunumeric/cudalibs.cu at line 88

Even if I set x=fx, y=fy, z=fz, it still crashes.

@magnatelee magnatelee self-assigned this Apr 14, 2022
@magnatelee
Copy link
Contributor

this should be fixed by #279. @eddy16112 can you pull my branch and see if that fixes the issue?

@magnatelee
Copy link
Contributor

leaving a note to myself and others: the issue was caused by the convolve implementation trying to call cuFFT with pointers that are not 8B aligned. This is actually a documented behavior: https://docs.nvidia.com/cuda/cufft/index.html#function-cufftexecr2c-cufftexecd2z Though this fix addresses the reported issue, I'll revert #204, which originally introduced this bug; while fixing the bug, it turns out that cuFFT callbacks make the convolution slower more than 3X for some unfortunate shapes, which is too great to justify the saving on memory space for temporary allocations.

@magnatelee
Copy link
Contributor

@eddy16112 Please pull and try again.

@eddy16112
Copy link
Author

It works for me. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants