Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor AsyncLLMEngine #880

Merged
merged 11 commits into from
Sep 4, 2023
Merged

Refactor AsyncLLMEngine #880

merged 11 commits into from
Sep 4, 2023

Conversation

Yard1
Copy link
Collaborator

@Yard1 Yard1 commented Aug 25, 2023

This PR aims to refactor and simplify AsyncLLMEngine implementation by moving to a single background scheduling loop. There should also be a tiny peformance boost.

@Yard1
Refactor AsyncLLMEngine
a2d5cc5 
Signed-off-by: Antoni Baum <[email protected]>
@Yard1
Lint
9107bc9 
Signed-off-by: Antoni Baum <[email protected]>
@Yard1
Rename
90ffbcc 
Signed-off-by: Antoni Baum <[email protected]>
@Yard1
Add ray_remote_kwargs
302d6fa 
Signed-off-by: Antoni Baum <[email protected]>
@Yard1
Nit
de30724 
Signed-off-by: Antoni Baum <[email protected]>
@Yard1
Merge branch 'vllm-project:main' into async_engine_refactor
b5de93e
@Yard1
Remove debug
e44fca4 
Signed-off-by: Antoni Baum <[email protected]>
@Yard1
Nit
9b1b34a 
Signed-off-by: Antoni Baum <[email protected]>
zhuohan123
zhuohan123 requested changes Aug 30, 2023
View reviewed changes
Copy link
Member

@zhuohan123 zhuohan123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! Please see the detailed comments below.

vllm/core/scheduler.py Outdated Show resolved Hide resolved
vllm/engine/async_llm_engine.py Outdated Show resolved Hide resolved
vllm/engine/async_llm_engine.py Outdated Show resolved Hide resolved
vllm/engine/async_llm_engine.py Show resolved Hide resolved
vllm/engine/llm_engine.py Outdated Show resolved Hide resolved
vllm/engine/llm_engine.py Outdated Show resolved Hide resolved
vllm/engine/llm_engine.py Outdated Show resolved Hide resolved
vllm/engine/llm_engine.py Outdated Show resolved Hide resolved
vllm/engine/async_llm_engine.py Outdated Show resolved Hide resolved
vllm/engine/async_llm_engine.py Outdated Show resolved Hide resolved
@zhuohan123 zhuohan123 mentioned this pull request Aug 30, 2023
Closed
@Yard1
Handle cancellations better
217d2b4 
Signed-off-by: Antoni Baum <[email protected]>
zhuohan123
zhuohan123 requested changes Sep 4, 2023
View reviewed changes
Copy link
Member

@zhuohan123 zhuohan123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes! Left some minor comments. Left some minor style comments. Let's merge the PR after these changes.

vllm/engine/async_llm_engine.py Outdated Show resolved Hide resolved
vllm/engine/async_llm_engine.py Outdated Show resolved Hide resolved
vllm/engine/async_llm_engine.py Outdated Show resolved Hide resolved
vllm/engine/llm_engine.py Outdated Show resolved Hide resolved
vllm/engine/llm_engine.py Outdated Show resolved Hide resolved
@esmeetu
Copy link
Collaborator

esmeetu commented Sep 4, 2023

FYI, Api server might stuck request when apply this PR.

@Yard1
Copy link
Collaborator Author

Yard1 commented Sep 4, 2023

@esmeetu can you elaborate?

zhuohan123
zhuohan123 approved these changes Sep 4, 2023
View reviewed changes
Copy link
Member

@zhuohan123 zhuohan123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you for your contribution!

@zhuohan123 zhuohan123 merged commit ce741ba into vllm-project:main Sep 4, 2023
2 checks passed
@Yard1 Yard1 deleted the async_engine_refactor branch September 4, 2023 04:45
@esmeetu
Copy link
Collaborator

esmeetu commented Sep 4, 2023

@Yard1
Start server: python -m vllm.entrypoints.openai.api_server --model llama-2-7b

POST /v1/completions
Content-Type: application/json;charset=UTF-8

{
"model": "llama-2-7b",
"prompt": [
"hi"
],
"max_tokens": 20,
"temperature": 0
}

Then no response...

@Yard1 Yard1 mentioned this pull request Sep 4, 2023
Merged
liuyanyi pushed a commit to liuyanyi/vllm that referenced this pull request Sep 12, 2023
@Yard1 @liuyanyi
Refactor AsyncLLMEngine (vllm-project#880)
9af27cb
hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024
@Yard1
Refactor AsyncLLMEngine (vllm-project#880)
bb98fb4
sjchoi1 pushed a commit to casys-kaist-internal/vllm that referenced this pull request May 7, 2024
@Yard1
Refactor AsyncLLMEngine (vllm-project#880)
b14c1b0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhuohan123 zhuohan123 zhuohan123 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Yard1 @esmeetu @zhuohan123