[Wait for #2568] [ Tensor ] add is_NaN check in Tensor @open sesame 05/10 14:17 #2574
Conversation
We will add Var32 Tensor if the Variable Weight is not Full precision (FP32). This eables the Weight Update with full precision and only Apply Gradient Process ueses this Tensor. Therefore, the lifespan of this tensor should be "ApplyGradient". . Modify TensorPool to generate Weigth considering Mixed Precsion. **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
This pr create the variable fp32 tensor when we create the Weight and Optimizer Weight. . update the manager to create Weight with var32 tensor which requested to weight pool. . update the weight requests with Weight Spec and var, grad and var32 tensors which created already. . add clone Tensor with specific type in tensor.h Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
This PR enables the FP16 support for the layers below: . input layer . mse loss layer Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
This PR includes the mixed precision test case. . Input - FC - MSE : "batch_size=2", "model_tensor_type=FP16-FP16", "loss_scale=128" **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
jijoongmoon
requested review from
myungjoo,
again4you,
jaeyun-jung,
leemgs,
wooksong,
helloahn,
kparichay,
gichan-jang,
anyj0527,
zhoonit,
lhs8928,
songgot,
jihochu,
DonghakPark,
SeoHyungjun,
baek2sm,
skykongkong8,
djeong20,
EunjuYang and
a team
as code owners
|
📝 TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2574. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/. |
|
|
cibot:
|
|
|
|
|
|
| @@ -1090,4 +1090,37 @@ void ele_div(const unsigned int N, const float *X, const float *Y, float *Z, | |||
| ele_div_fallback(N, X, Y, Z, alpha, beta, i_stride, o_stride); | |||
| } | |||
|
|
|||
| bool has_nan(const size_t N, ml::train::TensorDim::DataType d_type, | |||
There was a problem hiding this comment.
did not check for other wait-for PRs yet, but commit in THIS pr looks fine
This commit modify apply gradient in optimizer. We do not need to save optimizer variables in weight type. Only Optimizer needs the optimizer variables and we should update the weight with full precision to maintain the accuracy. Therefore, remove the var32 tensors for optimizer variables. Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
|
|
jijoongmoon
changed the title
|
|
|
|
|
|
jijoongmoon
force-pushed
the
is_nan
branch
2 times, most recently
from
b1dd77e
to
366c357
Compare
This PR add is_NaN function to check if the tensor has NaN value. This is for the check NaN during mixed precision training. **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
There was a problem hiding this comment.
@jijoongmoon, 💯 All CI checkers are successfully verified. Thanks.
| if get_option('enable-avx') | ||
| extra_defines += '-DUSE_AVX=1' | ||
| if get_option('platform') == 'tizen' | ||
| add_project_arguments(['-mavx2'], language: ['c','cpp']) |
There was a problem hiding this comment.
would the Tizen platform always support AVX2 instructions?
| int temp = 0; | ||
| size_t idx = 0; | ||
|
|
||
| // 16 single-precision check : ( X != X ) |
There was a problem hiding this comment.
| // 16 single-precision check : ( X != X ) | |
| // 16 half-precision check : ( X != X ) |
| return true; | ||
| } | ||
|
|
||
| // 8 single-precision check : ( X != X ) |
There was a problem hiding this comment.
| // 8 single-precision check : ( X != X ) | |
| // 8 half-precision check : ( X != X ) |
| %define avx_support -Denable-avx=true | ||
| %else | ||
| %define avx_support -Denable-avx=false | ||
| %endif # arch aarch64 |
There was a problem hiding this comment.
| %endif # arch aarch64 | |
| %endif # arch x86_64 |
|
closed by #2663 |
In this PR
This PR adds the is_NaN function to check if the tensor has a NaN value. This
is for the check of NaN during mixed precision training.
Self evaluation:
Signed-off-by: jijoong.moon [email protected]