Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Speculative Decoding] Support draft model on different tensor-parallel size than target model #5414

Merged
merged 131 commits into from
Jun 25, 2024

Commits on Jun 10, 2024

  1. tp1 draft worker

    wooyeonlee0 committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    f5b5f94 View commit details
    Browse the repository at this point in the history
  2. refactor singlt_tp_worker

    wooyeonlee0 committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    709de21 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. update execute_model logic

    minior
    
    add log
    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    0eacc96 View commit details
    Browse the repository at this point in the history
  2. fix

    log
    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    2011ed0 View commit details
    Browse the repository at this point in the history
  3. DummyProposerWorker

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    2e16c4e View commit details
    Browse the repository at this point in the history
  4. fix

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    b412a51 View commit details
    Browse the repository at this point in the history
  5. init only partial workers

    nit
    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    593ccfa View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    c5d3476 View commit details
    Browse the repository at this point in the history
  7. self._patch_tp_group

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    44e623b View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    98caf17 View commit details
    Browse the repository at this point in the history
  9. spec-tp configuarable

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    7fc4ff5 View commit details
    Browse the repository at this point in the history
  10. ngram worker support test

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    a96e720 View commit details
    Browse the repository at this point in the history
  11. minor refine

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    db39576 View commit details
    Browse the repository at this point in the history
  12. cleanup

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    b2e8595 View commit details
    Browse the repository at this point in the history
  13. return type fix

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    756442a View commit details
    Browse the repository at this point in the history
  14. cleanup

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    32094f1 View commit details
    Browse the repository at this point in the history
  15. cleanup

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    7890191 View commit details
    Browse the repository at this point in the history
  16. typo

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    53b2ea9 View commit details
    Browse the repository at this point in the history
  17. verify arg

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    a29c9c5 View commit details
    Browse the repository at this point in the history
  18. remove testing code

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    52ba09d View commit details
    Browse the repository at this point in the history
  19. cleanup

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    d26ef08 View commit details
    Browse the repository at this point in the history
  20. rename module

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    80c4994 View commit details
    Browse the repository at this point in the history
  21. cleanup

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    0f16f3f View commit details
    Browse the repository at this point in the history
  22. cleanup

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    140f478 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    3fd7e91 View commit details
    Browse the repository at this point in the history
  24. fix

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    495aa30 View commit details
    Browse the repository at this point in the history
  25. undo unrelated changes

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    3a5a47f View commit details
    Browse the repository at this point in the history
  26. minor fix

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    07ddbb8 View commit details
    Browse the repository at this point in the history
  27. fix ruff errors

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    b0a677d View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    96782a2 View commit details
    Browse the repository at this point in the history
  29. typo

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    9998b9c View commit details
    Browse the repository at this point in the history
  30. temporal fix

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    e92ecdc View commit details
    Browse the repository at this point in the history
  31. formatting

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    b421607 View commit details
    Browse the repository at this point in the history
  32. isort

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    386ab9b View commit details
    Browse the repository at this point in the history
  33. line length

    wooyeonlee0 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    b25f74e View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. fix

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    8b51f08 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d4b283c View commit details
    Browse the repository at this point in the history
  3. line length

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    dfc90cb View commit details
    Browse the repository at this point in the history
  4. comment

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    9bef5e4 View commit details
    Browse the repository at this point in the history
  5. add type hint

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    85d087d View commit details
    Browse the repository at this point in the history
  6. isort

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    9af36b7 View commit details
    Browse the repository at this point in the history
  7. add more type hints

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    5a0bf45 View commit details
    Browse the repository at this point in the history
  8. fix

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    531c9f0 View commit details
    Browse the repository at this point in the history
  9. test

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    287da20 View commit details
    Browse the repository at this point in the history
  10. nit

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    08d1b2a View commit details
    Browse the repository at this point in the history
  11. fix yapf

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    237c966 View commit details
    Browse the repository at this point in the history
  12. fix

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    0bb38c2 View commit details
    Browse the repository at this point in the history
  13. fix

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    c097d6c View commit details
    Browse the repository at this point in the history
  14. fix

    wooyeonlee0 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    957a325 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Configuration menu
    Copy the full SHA
    3ec8cb5 View commit details
    Browse the repository at this point in the history
  2. add comments

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    8a8a1e4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7f06f64 View commit details
    Browse the repository at this point in the history
  4. fix

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    1e87579 View commit details
    Browse the repository at this point in the history
  5. fix

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    abc546c View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    7880cb0 View commit details
    Browse the repository at this point in the history
  7. nit

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    2ebe6f3 View commit details
    Browse the repository at this point in the history
  8. fix

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    90d46ee View commit details
    Browse the repository at this point in the history
  9. refactor. remove log

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    7e1426c View commit details
    Browse the repository at this point in the history
  10. remove return

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    ad52d93 View commit details
    Browse the repository at this point in the history
  11. fix

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    355475b View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    9cfdb5b View commit details
    Browse the repository at this point in the history
  13. nit

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    6a6c5ff View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    ddef229 View commit details
    Browse the repository at this point in the history
  15. fix ruff errors

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    965f648 View commit details
    Browse the repository at this point in the history
  16. isort

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    1bb5534 View commit details
    Browse the repository at this point in the history
  17. fix yapf

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    ea6b8f5 View commit details
    Browse the repository at this point in the history
  18. undo ngramworker support

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    71977d2 View commit details
    Browse the repository at this point in the history
  19. add comment

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    bc5f77a View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    5655a49 View commit details
    Browse the repository at this point in the history
  21. ruff

    wooyeonlee0 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    eabc16a View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. remove ranks arg

    wooyeonlee0 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    f748edf View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c099c94 View commit details
    Browse the repository at this point in the history
  3. undo

    wooyeonlee0 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    4b74a45 View commit details
    Browse the repository at this point in the history
  4. add dist test

    wooyeonlee0 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    c9786ad View commit details
    Browse the repository at this point in the history
  5. nit

    wooyeonlee0 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    a42664a View commit details
    Browse the repository at this point in the history
  6. fix

    wooyeonlee0 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    ac7701a View commit details
    Browse the repository at this point in the history
  7. test fix

    wooyeonlee0 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    eea6a7e View commit details
    Browse the repository at this point in the history
  8. yapf fix

    wooyeonlee0 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    a648f5d View commit details
    Browse the repository at this point in the history
  9. update comment

    wooyeonlee0 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    f23ba8c View commit details
    Browse the repository at this point in the history
  10. require 2 gpus

    wooyeonlee0 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    aa9af93 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. Configuration menu
    Copy the full SHA
    56c8927 View commit details
    Browse the repository at this point in the history
  2. comment

    wooyeonlee0 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    385b4f8 View commit details
    Browse the repository at this point in the history
  3. ruff mypy

    wooyeonlee0 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    43f37eb View commit details
    Browse the repository at this point in the history
  4. isort

    wooyeonlee0 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    99350e2 View commit details
    Browse the repository at this point in the history
  5. yapf

    wooyeonlee0 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    a9f3e23 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    6ba250d View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    3e78613 View commit details
    Browse the repository at this point in the history
  8. yapf

    wooyeonlee0 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    6532af7 View commit details
    Browse the repository at this point in the history
  9. yapf

    wooyeonlee0 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    6839797 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. Configuration menu
    Copy the full SHA
    aac586b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    98e584d View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. add comment

    wooyeonlee0 committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    2d5e64d View commit details
    Browse the repository at this point in the history
  2. yapf

    wooyeonlee0 committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    ba88bd4 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. resolve cade comments

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    46e5274 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    85f4f25 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c1b5373 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4a58617 View commit details
    Browse the repository at this point in the history
  5. ruff, yapf

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    b09e7be View commit details
    Browse the repository at this point in the history
  6. remove world group patch

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    7168d78 View commit details
    Browse the repository at this point in the history
  7. isort, yapf

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    fe0bd5b View commit details
    Browse the repository at this point in the history
  8. yield fix

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    2e0d170 View commit details
    Browse the repository at this point in the history
  9. debugging

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    36f8aa5 View commit details
    Browse the repository at this point in the history
  10. log

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    54bf514 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    bfd7d2f View commit details
    Browse the repository at this point in the history
  12. add lora methods

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    f337428 View commit details
    Browse the repository at this point in the history
  13. missing method

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    4654b9f View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    e39926e View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    1c6eefd View commit details
    Browse the repository at this point in the history
  16. remove unused logger

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    f2d2ee5 View commit details
    Browse the repository at this point in the history
  17. isort. minor rename

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    302955c View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    3d4754e View commit details
    Browse the repository at this point in the history
  19. yapf, ruff

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    620b224 View commit details
    Browse the repository at this point in the history
  20. add skip_spec_test

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    b245d3c View commit details
    Browse the repository at this point in the history
  21. remove spec-tp 3 case

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    1e71e98 View commit details
    Browse the repository at this point in the history
  22. spec-draft-tp

    wooyeonlee0 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    a01c00d View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. _TP_STATE_PATCHED

    wooyeonlee0 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    debffc2 View commit details
    Browse the repository at this point in the history
  2. remove stale comment

    wooyeonlee0 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    39fe67f View commit details
    Browse the repository at this point in the history
  3. dist_tp2, dist_tp4 tests

    wooyeonlee0 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    af1b0be View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    834c6e0 View commit details
    Browse the repository at this point in the history
  5. comment

    wooyeonlee0 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    5bc2bc3 View commit details
    Browse the repository at this point in the history
  6. yapf

    wooyeonlee0 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    8740369 View commit details
    Browse the repository at this point in the history
  7. comment

    wooyeonlee0 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    4d82ca1 View commit details
    Browse the repository at this point in the history
  8. undo change in test utils

    wooyeonlee0 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    7bf831c View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    3fccc76 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. Configuration menu
    Copy the full SHA
    e8d0e93 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    91c2e43 View commit details
    Browse the repository at this point in the history
  3. yapf

    wooyeonlee0 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    fac7e68 View commit details
    Browse the repository at this point in the history
  4. add todo comment

    wooyeonlee0 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    271822e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    ae0d7f1 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b84a070 View commit details
    Browse the repository at this point in the history
  7. yapf

    wooyeonlee0 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    86fda24 View commit details
    Browse the repository at this point in the history