Skip to content

Commit

Permalink
add more clarity on spec and align with real implementation
Browse files Browse the repository at this point in the history
  • Loading branch information
yding10 committed Jun 1, 2021
1 parent f8c12a1 commit a3b232a
Showing 1 changed file with 8 additions and 5 deletions.
13 changes: 8 additions & 5 deletions docs/ops/activation/SoftPlus_4.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,19 @@ SoftPlus(x) = \left\{\begin{array}{r}
\end{array}\right.
\f]

**Note**: For numerical stability the operation reverts to the linear function when `x > threshold` where `threshold` depends on *T* and
is chosen in such a way that the difference between the linear function and exact calculation is no more than `1e-6`.
The `threshold` can be calculated with the following formula where `alpha` is the number of digits after the decimal point,
`beta` is maximum value of *T* data type:
**Note**:

For numerical stability the operation reverts to the linear function when `x > threshold` where `threshold` depends on *T* and is chosen in such a way that the difference between the linear function and exact calculation is no more than `1e-6`.

The `threshold` is internally defined parameter, which can be calculated with the following formula where `alpha` is the number of digits after the decimal point, `beta` is maximum value of *T* data type:

\f[
-log(e^{10^{-\alpha}} - 1.0) < threshold < log(\beta)
\f]

For example, if *T* is `fp32`, `threshold` should be `20` or if *T* is `fp16`, `threshold` should be `11`.
For example:
If *T* is `fp32`, `threshold` range is `13.8 ~ 88.7`, in practice `threshold` is set to be `20`.
If *T* is `fp16`, in practice `threshold` is also set to be `20`,which is also ok as the data is operated in `fp32` format and then converted to `fp16` at the end.

**Attributes**: *SoftPlus* operation has no attributes.

Expand Down

0 comments on commit a3b232a

Please sign in to comment.