Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Wait for #2574] [ Context ] Add loss scale in Context & using mse loss #2580

Closed
wants to merge 7 commits into from

Commits on May 7, 2024

  1. [ Weight ] Add Var32 Tensor in Weight.

    We will add Var32 Tensor if the Variable Weight is not Full
    precision (FP32). This eables the Weight Update with full precision
    and only Apply Gradient Process ueses this Tensor. Therefore, the
    lifespan of this tensor should be "ApplyGradient".
    
    . Modify TensorPool to generate Weigth considering Mixed Precsion.
    
    **Self evaluation:**
    1. Build test:	 [X]Passed [ ]Failed [ ]Skipped
    2. Run test:	 [X]Passed [ ]Failed [ ]Skipped
    
    Signed-off-by: jijoong.moon <[email protected]>
    jijoongmoon committed May 7, 2024
    Configuration menu
    Copy the full SHA
    a92784b View commit details
    Browse the repository at this point in the history
  2. [ Mixed ] Create weight with var32 tensor

    This pr create the variable fp32 tensor when we create the Weight and
    Optimizer Weight.
    
    . update the manager to create Weight with  var32 tensor which
    requested to weight pool.
    . update the weight requests with Weight Spec and var, grad and var32
    tensors which created already.
    . add clone Tensor with specific type in tensor.h
    
    Resolves:
    
    **Self evaluation:**
    1. Build test:	 [X]Passed [ ]Failed [ ]Skipped
    2. Run test:	 [X]Passed [ ]Failed [ ]Skipped
    
    Signed-off-by: jijoong.moon <[email protected]>
    jijoongmoon committed May 7, 2024
    Configuration menu
    Copy the full SHA
    b6ad1d0 View commit details
    Browse the repository at this point in the history
  3. [ Layers ] Update Layers to support FP16

    This PR enables the FP16 support for the layers below:
    
    . input layer
    . mse loss layer
    
    Resolves:
    
    **Self evaluation:**
    1. Build test:	 [X]Passed [ ]Failed [ ]Skipped
    2. Run test:	 [X]Passed [ ]Failed [ ]Skipped
    
    Signed-off-by: jijoong.moon <[email protected]>
    jijoongmoon committed May 7, 2024
    Configuration menu
    Copy the full SHA
    8b7b44f View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. [ Test ] Mixed Precision Test Case

    This PR includes the mixed precision test case.
    
    . Input - FC - MSE
     : "batch_size=2", "model_tensor_type=FP16-FP16", "loss_scale=128"
    
    **Self evaluation:**
    1. Build test:	 [X]Passed [ ]Failed [ ]Skipped
    2. Run test:	 [X]Passed [ ]Failed [ ]Skipped
    
    Signed-off-by: jijoong.moon <[email protected]>
    jijoongmoon committed May 8, 2024
    Configuration menu
    Copy the full SHA
    27ebec6 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. [ Optimizer ] Update Optimizer / Adam to support Mixed training

    This commit modify apply gradient in optimizer.
    We do not need to save optimizer variables in weight type. Only
    Optimizer needs the optimizer variables and we should update the
    weight with full precision to maintain the accuracy. Therefore,
    remove the var32 tensors for optimizer variables.
    
    Resolves:
    
    **Self evaluation:**
    1. Build test:	 [X]Passed [ ]Failed [ ]Skipped
    2. Run test:	 [X]Passed [ ]Failed [ ]Skipped
    
    Signed-off-by: jijoong.moon <[email protected]>
    jijoongmoon committed May 9, 2024
    Configuration menu
    Copy the full SHA
    0593c27 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. [ Tensor ] add is_NaN check in Tensor

    This PR add is_NaN function to check if the tensor has NaN value. This
    is for the check NaN during mixed precision training.
    
    **Self evaluation:**
    1. Build test:	 [X]Passed [ ]Failed [ ]Skipped
    2. Run test:	 [X]Passed [ ]Failed [ ]Skipped
    
    Signed-off-by: jijoong.moon <[email protected]>
    jijoongmoon committed May 10, 2024
    Configuration menu
    Copy the full SHA
    59b7c2e View commit details
    Browse the repository at this point in the history

Commits on May 11, 2024

  1. [ Context ] Add loss scale in Context & using mse loss

    This PR add loss scale parameter in runcontext and use it to update
    mse loss.
    
    . Add Loss Scale Parameter in RunLayerContext Constructor
    . Add applyLossScale func to update return derivitive in Loss Layer
    . Change MSE Loss Layer to apply the loss scale to return derivitive
    
    **Self evaluation:**
    1. Build test:	 [X]Passed [ ]Failed [ ]Skipped
    2. Run test:	 [X]Passed [ ]Failed [ ]Skipped
    
    Signed-off-by: jijoong.moon <[email protected]>
    jijoongmoon committed May 11, 2024
    Configuration menu
    Copy the full SHA
    adc2f2a View commit details
    Browse the repository at this point in the history