Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mirror PR: [Speculative decoding 6/9] Integrate speculative decoding with LLMEngine #3

Closed
wants to merge 127 commits into from
Closed

Mirror PR: [Speculative decoding 6/9] Integrate speculative decoding with LLMEngine #3

wants to merge 127 commits into from

Conversation

cadedaniel
Copy link
Owner

@cadedaniel cadedaniel commented Apr 7, 2024 •
edited
Loading

@cadedaniel cadedaniel changed the title Llm engine spec Mirror PR: [Speculative decoding 6/9] Integrate speculative decoding with LLMEngine Apr 7, 2024
rkooo567 and others added 29 commits April 12, 2024 14:35
@rkooo567
[mypy] Add mypy type annotation part 1 (vllm-project#4006)
09473ee
@youkaichao
[Core] fix custom allreduce default value (vllm-project#4040)
fbb9d9e
@Bellk17 @WoosukKwon
Fix triton compilation issue (vllm-project#3984)
d04973a 
Co-authored-by: Woosuk Kwon <[email protected]>
@jeejeelee
[Bugfix] Fix LoRA bug (vllm-project#4032)
b8aacac
@youkaichao
[CI/Test] expand ruff and yapf for all supported python version (vllm…
546e721 
…-project#4037)
@dylanwhawk
[Bugfix] More type hint fixes for py 3.8 (vllm-project#4039)
5c2e66e
@youkaichao
[Core][Distributed] improve logging for init dist (vllm-project#4042)
98afde1
@zspo
[Bugfix] fix_log_time_in_metrics (vllm-project#4050)
ec8e3c6
@zspo
[Bugfix] fix_small_bug_in_neuron_executor (vllm-project#4051)
0a430b4
@jeejeelee
[Kernel] Add punica dimension for Baichuan-13B (vllm-project#4053)
989ae25
@sangstar
[Frontend] [Core] feat: Add model loading using tensorizer (vllm-pr…
711a000 
…oject#3476)
@youkaichao
[Core] avoid too many cuda context by caching p2p test (vllm-project#…
2cd6b4f 
…4021)
@njhill
[BugFix] Fix tensorizer extra in setup.py (vllm-project#4072)
563c54f
@simon-mo
[Docs] document that mixtral 8x22b is supported (vllm-project#4073)
aceb17c
@esmeetu
[Misc] Upgrade triton to 2.2.0 (vllm-project#4061)
8db1bf3
@zhuohan123
[Bugfix] Fix filelock version requirement (vllm-project#4075)
e11e200
@bigPYJ1151
[Misc][Minor] Fix CPU block num log in CPUExecutor. (vllm-project#4088)
0003e91
@njhill
[Core] Simplifications to executor classes (vllm-project#4071)
eb46fbf
@sangstar @ywang96
[Doc] Add better clarity for tensorizer usage (vllm-project#4090)
d619ae2 
Co-authored-by: Roger Wang <[email protected]>
@rickyyx
[Bugfix] Fix ray workers profiling with nsight (vllm-project#4095)
4695397
@rkooo567
[Typing] Fix Sequence type GenericAlias only available after Python 3…
37e84a4 
….9. (vllm-project#4092)
@rkooo567
[Core] Fix engine-use-ray broken (vllm-project#4105)
4e7ee66
@noamgat @simon-mo
LM Format Enforcer Guided Decoding Support (vllm-project#3868)
0543476 
Co-authored-by: Simon Mo <[email protected]>
@cadedaniel
allow append empty tokens in block table
2a19f5e
@cadedaniel
Merge remote-tracking branch 'upstream/main' into llm-engine-spec
79325d3
@cadedaniel
rebase on stop string fixes
b6e9e82
@cadedaniel
test spec
bf0c37c
@cadedaniel
lint & mypy
a158256
@cadedaniel
doc
5a69f6c
@cadedaniel cadedaniel closed this Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.