Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revisit attention layer for fp16 test and apply template in activation function #28

Merged
merged 3 commits into from
Aug 10, 2023
Merged

Revisit attention layer for fp16 test and apply template in activation function #28

merged 3 commits into from
Aug 10, 2023

Conversation

skykongkong8
Copy link

@skykongkong8 skykongkong8 commented Aug 10, 2023 •
edited
Loading

Commits to be reviewd in this PR

Add dataset file for attention layer
  • Add new .nnlayergolden file unpacked when meson build

Signed-off-by:sungsik.kong[email protected]

Apply template on activation functions
  • As far as I am concerned, activation functions are better to be handled with template instantiation since we are planning to support multiple types
  • Unlike previous trial in Tensor template instantiation, here we get the layer activation datatype from runContext finalize function
  • Please note that there are some unverified layers w.r.t. fp16, and such layers might need revision due to this change. (this work will be continued in the next PR, but there are no conflicts for now)

Signed-off-by:sungsik.kong[email protected]

Verify Attention layer with fp16
  • On top of the previous commit, verified the attention layer.
  • Revision for other layers will be continued on top of this PR

Signed-off-by:sungsik.kong[email protected]

Self evaluation:

  1. Build test: [X]Passed [ ]Failed [ ]Skipped
  2. Run test: [X]Passed [ ]Failed [ ]Skipped

@skykongkong8
[gtest] Add dataset file for attention layer
c6076c3 
    * Now nnlayergolden binary file for attention layer gtest will be automatically generated when build

Signed-off-by: skykongkong8 <[email protected]>
@skykongkong8
[layers/activation_func] Apply template on activation functions
dde0727 
**Changes proposed in this PR:**

- For mixed precision, activation functions should be revised to a function template to avoid bulky code
- In order to use function template for setActivation, we need another function template to handle multiple types of activation function
- Minor fixes for template instantiation, and this will be revised proplerly for fp16 use in the next PR

Resolves:

**Self evaluation:**
1. Build test:     [X]Passed [ ]Failed [ ]Skipped
2. Run test:     [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: sungsik.kong <[email protected]>
@skykongkong8
[gtest] Verify attention layer with fp16
6710963 
- Add fp16 test case
- Modify epsilon value in cosine similarity with proper decimal number & significant digit

Resolves:

**Self evaluation:**
1. Build test:     [X]Passed [ ]Failed [ ]Skipped
2. Run test:     [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: sungsik.kong <[email protected]>
jijoongmoon
jijoongmoon approved these changes Aug 10, 2023
View reviewed changes
Copy link
Owner

@jijoongmoon jijoongmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jijoongmoon jijoongmoon merged commit dade59b into jijoongmoon:tensor_type_in_dim Aug 10, 2023
7 checks passed
@skykongkong8 skykongkong8 deleted the jjm_tensor_type_in_dim branch February 14, 2024 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jijoongmoon jijoongmoon jijoongmoon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@skykongkong8 @jijoongmoon