-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Fix] Fix get_valid_count flaky test for cuda #4901
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for fixing this!
ping @kevinthesun @yzhliu |
* get_valid_count accuracy issue fixed for individual tests but not for all tests running together * minor fix * initialize valid_count and PrefixSum buffers * test updated * udpate relay test as well * update document * fix lint * address comment * fix lint * correct atomicAdd identifier name
* get_valid_count accuracy issue fixed for individual tests but not for all tests running together * minor fix * initialize valid_count and PrefixSum buffers * test updated * udpate relay test as well * update document * fix lint * address comment * fix lint * correct atomicAdd identifier name
* get_valid_count accuracy issue fixed for individual tests but not for all tests running together * minor fix * initialize valid_count and PrefixSum buffers * test updated * udpate relay test as well * update document * fix lint * address comment * fix lint * correct atomicAdd identifier name
@Laurawly I still get a flaky failure from get_valid_count test in my PR |
@masahi That’s weird. You can comment off the test for now and I’ll try to reproduce it on my end. |
@Laurawly Also reported at a different PR #4931 (comment) |
Yeah, pls feel free to comment out the test: https://github.com/apache/incubator-tvm/blob/master/topi/tests/python/test_topi_vision.py#L106 |
fortunately my PR is green now without commenting out :) If the next open PR by somebody else sees the same problem, I'll ask him/her to comment it out. |
Turned on get_valid_count test for cuda in topi. Used atomic operations in this fix to replace previous block sync method.
Tested on V100 and T4 GPUs.
@trevor-m @kevinthesun @yzhliu Could you review?