New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

习题 8-2 #77

Open

Jasonmils opened this issue Aug 29, 2022 · 0 comments

Jasonmils commented Aug 29, 2022

输入x的方差越大， exp(x)的方差就越大，所以softmax后得到的向量分布更加趋近于0或者1，softmax的梯度如下

因此其梯度更加趋近于0

缩小x的方差可以限制softmax的两级（01）分化趋势，从而减缓梯度消失

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment