-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add node-wise normalization mode in LayerNorm
#4944
Conversation
Codecov Report
@@ Coverage Diff @@
## master #4944 +/- ##
==========================================
- Coverage 84.65% 82.71% -1.95%
==========================================
Files 330 330
Lines 17885 17887 +2
==========================================
- Hits 15141 14795 -346
- Misses 2744 3092 +348
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Super. Only left nitpicky comments :D
Co-authored-by: Matthias Fey <[email protected]>
Co-authored-by: Matthias Fey <[email protected]>
Co-authored-by: Matthias Fey <[email protected]>
for more information, see https://pre-commit.ci
Co-authored-by: Matthias Fey <[email protected]>
Co-authored-by: Matthias Fey <[email protected]>
Co-authored-by: Matthias Fey <[email protected]>
Co-authored-by: Matthias Fey <[email protected]>
for more information, see https://pre-commit.ci
Thanks! Looks cleaner. |
This pull request adds the node-wise normalization mode in
LayerNorm
. If"graph"
mode is used (by default), each graph will be considered as an element to be normalized. If"node"
mode is used, each node will be considered as an element to be normalized.