Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core][Bugfix] Support prompt_logprobs returned with speculative decoding #8047

Merged
merged 9 commits into from
Sep 25, 2024
Merged

[Core][Bugfix] Support prompt_logprobs returned with speculative decoding #8047

merged 9 commits into from
Sep 25, 2024

Commits on Sep 18, 2024

  1. fix: use INVALID_TOKEN_ID instead of uninitialized tensor data 

    Signed-off-by: Travis Johnson <[email protected]>
    @tjohnson31415
    tjohnson31415 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    488240d View commit details
    Browse the repository at this point in the history

  2. refactor: use INVALID_TOKEN_ID instead of magic number 

    Signed-off-by: Travis Johnson <[email protected]>
    @tjohnson31415
    tjohnson31415 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    fb58cd4 View commit details
    Browse the repository at this point in the history

  3. fix: skip entries for prompt tokens in spec decoding 

    Signed-off-by: Travis Johnson <[email protected]>
    @tjohnson31415
    tjohnson31415 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    9bcf56b View commit details
    Browse the repository at this point in the history

  4. refactor: move INVALID_TOKEN_ID to sequence.py and rename 

    Signed-off-by: Travis Johnson <[email protected]>
    @tjohnson31415
    tjohnson31415 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    526f14c View commit details
    Browse the repository at this point in the history

  5. fix: detokenize negative id to empty string 

    Signed-off-by: Travis Johnson <[email protected]>
    @tjohnson31415
    tjohnson31415 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    041e5fa View commit details
    Browse the repository at this point in the history

  6. feat: support prompt_logprobs output with spec decoding 

    Signed-off-by: Travis Johnson <[email protected]>
    @tjohnson31415
    tjohnson31415 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    ad367ff View commit details
    Browse the repository at this point in the history

  7. test: allow generate_w_logprobs even if no logprobs requested 

    Signed-off-by: Travis Johnson <[email protected]>
    @tjohnson31415
    tjohnson31415 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    6f420db View commit details
    Browse the repository at this point in the history

  8. test: update spec_decode e2e tests 

    Include logprobs cases and where disable_logprobs is True.

Signed-off-by: Travis Johnson <[email protected]>
    @tjohnson31415
    tjohnson31415 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    a15fea6 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. changes from code review 

    Signed-off-by: Travis Johnson <[email protected]>
    @tjohnson31415
    tjohnson31415 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    6e8e54d View commit details
    Browse the repository at this point in the history