add other 15 activation ops #32622
Merged
add other 15 activation ops #32622
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
ZzSean
force-pushed
the
activation_op_impl
branch
from
39644c4 to
92761cf
Compare
ZzSean
force-pushed
the
activation_op_impl
branch
from
92761cf to
85104d6
Compare
Xreki
reviewed
Apr 30, 2021
| __device__ __forceinline__ T operator()(const T* args) const { | ||
| MPType x = static_cast<MPType>(args[0]); | ||
| MPType t = static_cast<MPType>(threshold); | ||
| MPType temp = (x > -t && x < t) ? x : (x <= -t ? -t : t); |
There was a problem hiding this comment.
同上，这种写法会需要2次、3次比较。可以看看：
MPType min = x < t ? x : t;
MPType max = min > -t ? min : -t;不过可能性能也没什么差别。
| __device__ __forceinline__ T operator()(const T* args) const { | ||
| MPType dout = static_cast<MPType>(args[0]); | ||
| MPType out = static_cast<MPType>(args[1]); | ||
| MPType t = static_cast<MPType>(threshold); |
There was a problem hiding this comment.
所以这个算子也是要求threshold > 0的吧？不知道OP实现里面有没有检查，或者加个注释说明下？
There was a problem hiding this comment.
加了注释，但是我看在api层面好像没有对threshold的正负做限制，感觉也需要加一下，不然为负值时，结果会不确定。
| MPType temp_x_neg = static_cast<MPType>(x <= zero); | ||
| return static_cast<T>( | ||
| dout * (temp_a_pos * temp_x_pos + temp_a_pos * temp_x_neg * a * exp(x) + | ||
| temp_a_neg * temp_x_pos * (one + a * exp(x)))); |
| // dx = 0, when x <= -3 | ||
| // dout , when x >= 3 | ||
| // dout * (x / 3 + 0.5), otherwise | ||
| // Inputs: args[0], the input dout |
There was a problem hiding this comment.
公式里面使用threshold、offset、scale。貌似python端没把这个几个参数暴露出来，但是op用了这几个参数。
ZzSean
force-pushed
the
activation_op_impl
branch
from
a25f611 to
1fdb9dc
Compare
Xreki
approved these changes
May 7, 2021
| return {{"threshold", &threshold}}; | ||
| } | ||
|
|
||
| // hadrshrink(x) = (x > -threshold && x < threshold) ? 0 : x |
| } | ||
|
|
||
| // hard_sigmoid(x) = 0, when x <= -3 | ||
| // 1, when x >= 3 |
There was a problem hiding this comment.
代码里面是和0、1比较，公式里面是和-3、3比较，是slope=6、offset=-3？
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Performance optimization
PR changes
OPs
Describe
Unify the implementation of activation operation
本次提交共修改激活算子16个，包括其前向和反向，其中
log 类算子 3 个: log1p, log2, log10
relu 类算子 5 个：brelu, soft_relu, relu6, thresholded_relu, elu
其他 8 个 ：stanh, soft_plus, soft_sign, tanh_shrink, hard_shrink, hard_sigmoid, swish, hard_swish
每种类型算子的性能提升近似，因此选取每个类别中的一个算子作为示例进行描述，如下表：
case配置：[16, 128, 257, 257]