Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Speculative decoding 6/9] Integrate speculative decoding with LLMEngine #3894

Merged
merged 120 commits into from
Apr 16, 2024
Merged

[Speculative decoding 6/9] Integrate speculative decoding with LLMEngine #3894

merged 120 commits into from
Apr 16, 2024

Commits on Apr 3, 2024

  1. wip

    @cadedaniel
    cadedaniel committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    252a0c7 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into executor_base

    @cadedaniel
    cadedaniel committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    dd629d4 View commit details
    Browse the repository at this point in the history

  3. wip

    @cadedaniel
    cadedaniel committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    a34800f View commit details
    Browse the repository at this point in the history

  4. wip

    @cadedaniel
    cadedaniel committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    09f30bd View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. clean

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    8b5bb8b View commit details
    Browse the repository at this point in the history

  2. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    6fd424f View commit details
    Browse the repository at this point in the history

  3. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    2a347bb View commit details
    Browse the repository at this point in the history

  4. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    658ff9b View commit details
    Browse the repository at this point in the history

  5. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    acee7be View commit details
    Browse the repository at this point in the history

  6. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    85760d6 View commit details
    Browse the repository at this point in the history

  7. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    408b29d View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'upstream/main' into executor_base

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    9d8fd69 View commit details
    Browse the repository at this point in the history

  9. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    3149a03 View commit details
    Browse the repository at this point in the history

  10. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    0c32e0a View commit details
    Browse the repository at this point in the history

  11. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    f64d5b1 View commit details
    Browse the repository at this point in the history

  12. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    7207f0c View commit details
    Browse the repository at this point in the history

  13. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    0c4df0b View commit details
    Browse the repository at this point in the history

  14. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    2e355e7 View commit details
    Browse the repository at this point in the history

  15. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    edb7f62 View commit details
    Browse the repository at this point in the history

  16. wip

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    48bb3e9 View commit details
    Browse the repository at this point in the history

  17. fix test

    @cadedaniel
    cadedaniel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    7b39044 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2024

  1. fix test

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    9e5f2fb View commit details
    Browse the repository at this point in the history

  2. fix test

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    1a3e26e View commit details
    Browse the repository at this point in the history

  3. fix test

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    cd2015c View commit details
    Browse the repository at this point in the history

  4. fix

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    d926034 View commit details
    Browse the repository at this point in the history

  5. fix

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    607f7e2 View commit details
    Browse the repository at this point in the history

  6. fix

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    e127bb7 View commit details
    Browse the repository at this point in the history

  7. fix

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    deaa8b0 View commit details
    Browse the repository at this point in the history

  8. clean

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    7817d61 View commit details
    Browse the repository at this point in the history

  9. clean

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    99823a3 View commit details
    Browse the repository at this point in the history

  10. fix

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    849bfe9 View commit details
    Browse the repository at this point in the history

  11. fix

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    951ba85 View commit details
    Browse the repository at this point in the history

  12. speed up cpu test

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    38948df View commit details
    Browse the repository at this point in the history

  13. wip

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    397ec77 View commit details
    Browse the repository at this point in the history

  14. wip

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    23382b9 View commit details
    Browse the repository at this point in the history

  15. clean

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    7a0294c View commit details
    Browse the repository at this point in the history

  16. wip

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    dcdca68 View commit details
    Browse the repository at this point in the history

  17. remove

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    ed58af2 View commit details
    Browse the repository at this point in the history

  18. Revert "more test speedup" 

    This reverts commit 4c486f9bb4fc3b90efc1765ba46f4a666d1c9339.
    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    df8688e View commit details
    Browse the repository at this point in the history

  19. wip

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    55a5203 View commit details
    Browse the repository at this point in the history

  20. wip

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    55d083b View commit details
    Browse the repository at this point in the history

  21. wip

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    0814d24 View commit details
    Browse the repository at this point in the history

  22. rename profile_num_available_blocks to get_max_allowed_kv_blocks

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    b18d00c View commit details
    Browse the repository at this point in the history

  23. rename again

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    8fb7b9a View commit details
    Browse the repository at this point in the history

  24. rename

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    3bb9e6f View commit details
    Browse the repository at this point in the history

  25. wip

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    edad09c View commit details
    Browse the repository at this point in the history

  26. wip

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    f93c845 View commit details
    Browse the repository at this point in the history

  27. wip

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    d2d2218 View commit details
    Browse the repository at this point in the history

  28. lint

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    2f960e7 View commit details
    Browse the repository at this point in the history

  29. wip

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    68552e1 View commit details
    Browse the repository at this point in the history

  30. import order

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    42983ba View commit details
    Browse the repository at this point in the history

  31. fix

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    2d5dbb8 View commit details
    Browse the repository at this point in the history

  32. docstrings

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    ae2f7e6 View commit details
    Browse the repository at this point in the history

  33. Merge branch 'main' into executor_base

    @cadedaniel
    cadedaniel authored Apr 5, 2024
    Configuration menu
    Copy the full SHA
    c89bb75 View commit details
    Browse the repository at this point in the history

  34. Merge remote-tracking branch 'upstream/main' into llm-engine-spec

    @cadedaniel
    cadedaniel committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    bf041d9 View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2024

  1. wip

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    fa8705d View commit details
    Browse the repository at this point in the history

  2. wip

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    8495321 View commit details
    Browse the repository at this point in the history

  3. wip

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    b63975b View commit details
    Browse the repository at this point in the history

  4. wip

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    cb23e8c View commit details
    Browse the repository at this point in the history

  5. wip

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    143ca28 View commit details
    Browse the repository at this point in the history

  6. fix

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    d8d4725 View commit details
    Browse the repository at this point in the history

  7. wip

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    b2728e0 View commit details
    Browse the repository at this point in the history

  8. assertion

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    6250f6c View commit details
    Browse the repository at this point in the history

  9. fix

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    a930755 View commit details
    Browse the repository at this point in the history

  10. fix

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    5b896a3 View commit details
    Browse the repository at this point in the history

  11. lint

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    bb43b53 View commit details
    Browse the repository at this point in the history

  12. fix

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    cde3160 View commit details
    Browse the repository at this point in the history

  13. fix

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    dd8aeff View commit details
    Browse the repository at this point in the history

  14. test

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    46e4847 View commit details
    Browse the repository at this point in the history

  15. test fixes

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    8454edc View commit details
    Browse the repository at this point in the history

  16. lint

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    819e656 View commit details
    Browse the repository at this point in the history

  17. Merge remote-tracking branch 'upstream/main' into executor_base

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    2b0d787 View commit details
    Browse the repository at this point in the history

  18. Merge remote-tracking branch 'upstream/main' into llm-engine-spec

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    67fd287 View commit details
    Browse the repository at this point in the history

  19. Merge branch 'executor_base' into llm-engine-spec

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    c3449ba View commit details
    Browse the repository at this point in the history

  20. clean

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    d0fbe47 View commit details
    Browse the repository at this point in the history

  21. refactor out beam search model processor

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    5445af6 View commit details
    Browse the repository at this point in the history

  22. fix

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    632b439 View commit details
    Browse the repository at this point in the history

  23. dedup stop check

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    26e7368 View commit details
    Browse the repository at this point in the history

  24. wip

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    06e7c01 View commit details
    Browse the repository at this point in the history

  25. del

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    184a52c View commit details
    Browse the repository at this point in the history

  26. rename

    @cadedaniel
    cadedaniel committed Apr 7, 2024
    Configuration menu
    Copy the full SHA
    34468fe View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2024

  1. wip

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    208c467 View commit details
    Browse the repository at this point in the history

  2. wip

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    3c6abcc View commit details
    Browse the repository at this point in the history

  3. wip

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    bbbcef7 View commit details
    Browse the repository at this point in the history

  4. fix

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    b58762d View commit details
    Browse the repository at this point in the history

  5. wip

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    8b500d4 View commit details
    Browse the repository at this point in the history

  6. unit tests for block decode

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    782ce22 View commit details
    Browse the repository at this point in the history

  7. stop token ids

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    3062e1c View commit details
    Browse the repository at this point in the history

  8. format

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    fba3b30 View commit details
    Browse the repository at this point in the history

  9. fixing spec tests

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    bda141f View commit details
    Browse the repository at this point in the history

  10. lint

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    49865fb View commit details
    Browse the repository at this point in the history

  11. clean up gpu executor

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    1a17ed1 View commit details
    Browse the repository at this point in the history

  12. wip

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    dea67bb View commit details
    Browse the repository at this point in the history

  13. fix

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    189d7eb View commit details
    Browse the repository at this point in the history

  14. wip

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    a70a040 View commit details
    Browse the repository at this point in the history

  15. detokenization

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    3e1b8f5 View commit details
    Browse the repository at this point in the history

  16. lint

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    b9777a6 View commit details
    Browse the repository at this point in the history

  17. docstrings

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    29b4f12 View commit details
    Browse the repository at this point in the history

  18. fix

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    42aa0bc View commit details
    Browse the repository at this point in the history

  19. more spec test

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    0ebd93b View commit details
    Browse the repository at this point in the history

  20. remove

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    33a3d72 View commit details
    Browse the repository at this point in the history

  21. wip

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    15c942d View commit details
    Browse the repository at this point in the history

  22. strip

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    063e34b View commit details
    Browse the repository at this point in the history

  23. print

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    672a855 View commit details
    Browse the repository at this point in the history

  24. fix flaky test

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    8021b38 View commit details
    Browse the repository at this point in the history

  25. reduce output len

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    8e93fff View commit details
    Browse the repository at this point in the history

  26. strip

    @cadedaniel
    cadedaniel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    d06e9a4 View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. pr feedback

    @cadedaniel
    cadedaniel committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    ca516aa View commit details
    Browse the repository at this point in the history

  2. Merge branch 'executor_base' into llm-engine-spec

    @cadedaniel
    cadedaniel committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    91cf0fc View commit details
    Browse the repository at this point in the history

  3. Zhuohan offline pr feedback

    @cadedaniel
    cadedaniel committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    f6c7b2e View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'upstream/main' into llm-engine-spec

    @cadedaniel
    cadedaniel committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    0283fae View commit details
    Browse the repository at this point in the history

  5. lint

    @cadedaniel
    cadedaniel committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    96f81c4 View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. pr feedback

    @cadedaniel
    cadedaniel committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    de16919 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. Merge branch 'main' into llm-engine-spec

    @cadedaniel
    cadedaniel authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    d933e50 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. allow append empty tokens in block table

    @cadedaniel
    cadedaniel committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    2a19f5e View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into llm-engine-spec

    @cadedaniel
    cadedaniel committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    79325d3 View commit details
    Browse the repository at this point in the history

  3. rebase on stop string fixes

    @cadedaniel
    cadedaniel committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    b6e9e82 View commit details
    Browse the repository at this point in the history

  4. test spec

    @cadedaniel
    cadedaniel committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    bf0c37c View commit details
    Browse the repository at this point in the history

  5. lint & mypy

    @cadedaniel
    cadedaniel committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    a158256 View commit details
    Browse the repository at this point in the history

  6. doc

    @cadedaniel
    cadedaniel committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    5a69f6c View commit details
    Browse the repository at this point in the history