forked from mlc-ai/mlc-llm
-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Enable Logprobs in MLC Batch Serving (#82)
* Squashed commit for logprobs implementation. Co-authored-by: Valery Chernov <[email protected]> Co-authored-by: Ilya Kozulin <[email protected]> * fix None check * Change detokenization to using token ids. * Fix wrong usage of token ids. Remove logging. * extend benchmarks for logprobs * fix test without logprobs * clean code * black format engine_common.py * logprobs is strictly bool, top_logprobs is int * refactor logprob info collection to not reduce performance * quick fix for check * review fix * fix list index out of range * rollback after rebase * test * Squashed commit for logprobs implementation. Co-authored-by: Valery Chernov <[email protected]> Co-authored-by: Ilya Kozulin <[email protected]> * fix None check * Change detokenization to using token ids. * Fix wrong usage of token ids. Remove logging. * extend benchmarks for logprobs * fix test without logprobs * clean code * black format engine_common.py * logprobs is strictly bool, top_logprobs is int * refactor logprob info collection to not reduce performance * quick fix for check * review fix * fix list index out of range * rollback after rebase * test * small fix * rename for the sake of clarity * some fixes with cpu-gpu tensor copying * refactor logprob pass to calculate * remove excess deps for token detokenization * small clean * small clean * return None instead of list of Nones * fix mypy --------- Co-authored-by: Valery Chernov <[email protected]> Co-authored-by: Ilya Kozulin <[email protected]> Co-authored-by: Valery Chernov <[email protected]>
- Loading branch information
1 parent
4535ff5
commit 2b3fcf0
Showing
22 changed files
with
376 additions
and
53 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.