[Misc] [Core] Implement RFC "Augment BaseExecutor interfaces to enable hardware-agnostic speculative decoding" #3837
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cadedaniel
changed the title
vllm/worker/cpu_worker.py
Outdated
| # Note: To reuse the cache management procedure, | ||
| # use cpu cache as 'gpu cache'. | ||
| num_cpu_blocks = num_gpu_blocks | ||
| del num_gpu_blocks |
There was a problem hiding this comment.
Confused about line 209-212, why del here? self.cache_config.num_gpu_blocks = num_gpu_blocks?
There was a problem hiding this comment.
I'm trying to make the code readable given the awkwardness of num_gpu_blocks actually being num_cpu_blocks and num_cpu_blocks being ignored/always zero. But yeah the del is a bit extreme for this..
I refactored out the checks; code is more readable now
| self.cache_config.num_gpu_blocks = num_cpu_blocks | ||
| self.cache_config.num_cpu_blocks = 0 | ||
|
|
||
| if num_cpu_blocks <= 0: |
There was a problem hiding this comment.
Nit: move the check before we read num_cpu_blocks
LiuXiaoxuanPKU
approved these changes
Apr 9, 2024
SageMoore
pushed a commit
to neuralmagic/nm-vllm
that referenced
this pull request
Apr 11, 2024
…e hardware-agnostic speculative decoding" (vllm-project#3837)
andy-neuma
pushed a commit
to neuralmagic/nm-vllm
that referenced
this pull request
Apr 12, 2024
…e hardware-agnostic speculative decoding" (vllm-project#3837)
This was referenced Apr 12, 2024
z103cb
pushed a commit
to z103cb/opendatahub_vllm
that referenced
this pull request
Apr 22, 2024
…e hardware-agnostic speculative decoding" (vllm-project#3837)
Temirulan
pushed a commit
to Temirulan/vllm-whisper
that referenced
this pull request
Sep 6, 2024
…e hardware-agnostic speculative decoding" (vllm-project#3837)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements the BaseWorker interface described in #3809. This enables speculative decoding to treat all workers in the same way, allowing future work to enable speculative decoding for CPU/Neuron/other vLLM backends.
I will write up docs on how to do this after spec decode is merged; the TL;DR is need to implement the rejection sampler (currently implemented only in pytorch) and add plumbing between the proposal method and verification model that works for the hardware backend (currently only top1 fixed speculation is implemented for torch).
Notes
Closes #3809