Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Relay] Improve numerical gradient check #3856

Merged
merged 1 commit into from
Aug 31, 2019
Merged

Conversation

SWu
Copy link
Contributor

@SWu SWu commented Aug 29, 2019

I was having trouble getting passing tests with the existing gradient check implementation, even though the gradient implementations match results from the keras framework. This change rewrites the gradient check helper to perform two-sided numerical checks on every gradient dimension. I have an upcoming pull request with several gradient implementations that I was able to verify with this implementation.

@SWu
Copy link
Contributor Author

SWu commented Aug 29, 2019

@MarisaKirisame @vinx13 @junrushao1994 please review

@SWu SWu mentioned this pull request Aug 29, 2019
if inputs is None:
params = fwd_func.params
# Generate random inputs on the same scale as epsilon to avoid numerical precision loss.
inputs = [_np_randn_from_type(x.checked_type, scale=(10 * eps)) for x in params]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In practice, I found that this heuristic works well for operations that involve multiplication/addition

For division you still get precision problems, especially when using float32. In those cases, either supply manually selected inputs, or use float64.

Copy link
Member

@vinx13 vinx13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I verified that conv2d grad can pass numerical test now

Copy link
Contributor

@MarisaKirisame MarisaKirisame left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vinx13 vinx13 merged commit a711f38 into apache:master Aug 31, 2019
@SWu SWu deleted the grad_check_fix branch September 2, 2019 15:50
wweic pushed a commit to wweic/tvm that referenced this pull request Sep 16, 2019
wweic pushed a commit to wweic/tvm that referenced this pull request Sep 16, 2019
wweic pushed a commit to neo-ai/tvm that referenced this pull request Sep 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants