Update IPEX_LLM_PERFORMANCE_MODE with input length threshold
#11908
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oscilloscope98
changed the title
IPEX_LLM_PERFORMANCE_MODE with input length threshold
jason-dai
reviewed
Aug 23, 2024
Oscilloscope98
commented
Aug 23, 2024
| if inputs_embeds.shape[1] >= PERFORMANCE_MODE_LOOKUP_INPUT_THRESHOLD: | ||
| lookahead = 2 # default to 2 now | ||
| else: | ||
| lookahead = 0 |
There was a problem hiding this comment.
We set lookahead=0 when meeting threshold but not directly use original generate function for performance mode benchmarking purposes.
There was a problem hiding this comment.
May have some further verification whether this way will bring too much overhead.
jason-dai
reviewed
Aug 23, 2024
Comment on lines
67
to
69
| else: | ||
| lookahead = 0 | ||
| use_update_candidate_strategy = False |
There was a problem hiding this comment.
Just set lookahead to None for now
There was a problem hiding this comment.
Have updated :) But then the all-in-one benchmark may not work for now with input_len < 100 for performance mode
|
Merge for now :) Will try to update the all-in-one benchmark tool in another PR accordingly next Monday. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Update IPEX_LLM_PERFORMANCE_MODE with input length threshold.