You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently those methods are automatically using he_uniform, with assumed scale of 1. This causes problems on large shaped objects, e.g. at some input / outputs I get scale of 0.005 for the uniform method, which causes the network to misbehave and dead neurons to appear (gradients close/equal to 0).
There's no other trivial way to change the initialization methodology except editing the library code.
The text was updated successfully, but these errors were encountered:
Allow passing of an opt parameter for variable initialization (scale) in the conv1d, aconv1d, embed, etc. methods (can be found here: https://github.com/buriburisuri/sugartensor/blob/master/sugartensor/sg_layer.py).
Currently those methods are automatically using he_uniform, with assumed scale of 1. This causes problems on large shaped objects, e.g. at some input / outputs I get scale of 0.005 for the uniform method, which causes the network to misbehave and dead neurons to appear (gradients close/equal to 0).
There's no other trivial way to change the initialization methodology except editing the library code.
The text was updated successfully, but these errors were encountered: