-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Wait for #2574] [ Context ] Add loss scale in Context & using mse loss #2580
Commits on May 7, 2024
-
[ Weight ] Add Var32 Tensor in Weight.
We will add Var32 Tensor if the Variable Weight is not Full precision (FP32). This eables the Weight Update with full precision and only Apply Gradient Process ueses this Tensor. Therefore, the lifespan of this tensor should be "ApplyGradient". . Modify TensorPool to generate Weigth considering Mixed Precsion. **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for a92784b - Browse repository at this point
Copy the full SHA a92784bView commit details -
[ Mixed ] Create weight with var32 tensor
This pr create the variable fp32 tensor when we create the Weight and Optimizer Weight. . update the manager to create Weight with var32 tensor which requested to weight pool. . update the weight requests with Weight Spec and var, grad and var32 tensors which created already. . add clone Tensor with specific type in tensor.h Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for b6ad1d0 - Browse repository at this point
Copy the full SHA b6ad1d0View commit details -
[ Layers ] Update Layers to support FP16
This PR enables the FP16 support for the layers below: . input layer . mse loss layer Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 8b7b44f - Browse repository at this point
Copy the full SHA 8b7b44fView commit details
Commits on May 8, 2024
-
[ Test ] Mixed Precision Test Case
This PR includes the mixed precision test case. . Input - FC - MSE : "batch_size=2", "model_tensor_type=FP16-FP16", "loss_scale=128" **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 27ebec6 - Browse repository at this point
Copy the full SHA 27ebec6View commit details
Commits on May 9, 2024
-
[ Optimizer ] Update Optimizer / Adam to support Mixed training
This commit modify apply gradient in optimizer. We do not need to save optimizer variables in weight type. Only Optimizer needs the optimizer variables and we should update the weight with full precision to maintain the accuracy. Therefore, remove the var32 tensors for optimizer variables. Resolves: **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 0593c27 - Browse repository at this point
Copy the full SHA 0593c27View commit details
Commits on May 10, 2024
-
[ Tensor ] add is_NaN check in Tensor
This PR add is_NaN function to check if the tensor has NaN value. This is for the check NaN during mixed precision training. **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 59b7c2e - Browse repository at this point
Copy the full SHA 59b7c2eView commit details
Commits on May 11, 2024
-
[ Context ] Add loss scale in Context & using mse loss
This PR add loss scale parameter in runcontext and use it to update mse loss. . Add Loss Scale Parameter in RunLayerContext Constructor . Add applyLossScale func to update return derivitive in Loss Layer . Change MSE Loss Layer to apply the loss scale to return derivitive **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for adc2f2a - Browse repository at this point
Copy the full SHA adc2f2aView commit details