[BugFix]: fix engine timeout due to request abort #6255

pushan01 · 2024-07-09T09:37:11Z

In async llm engine, The request will be aborted after finished (async_llm_engine.py:L135), but the engine_step may still return True due to the difference between request finish and engine generate finish judgement. If the engine request is aborted but the engine_step return True, the step would be residual till next request arrive, which makes engine die due to timeout error if the next request arrive after ENGINE_ITERATION_TIMEOUT_S.

This PR fixed this problem by keep up with the engine_step and the request finish.

FILL IN THE PR DESCRIPTION HERE

FIX #6254 (link existing issues this PR will resolve)

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

PR Checklist (Click to Expand)

Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

[Bugfix] for bug fixes.
[CI/Build] for build or continuous integration improvements.
[Doc] for documentation fixes and improvements.
[Model] for adding a new model or improving an existing model. Model name should appear in the title.
[Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
[Kernel] for changes affecting CUDA kernels or other compute kernels.
[Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
[Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
[Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

We adhere to Google Python style guide and Google C++ style guide.
Pass all linter checks. Please use format.sh to format your code.
The code need to be well-documented to ensure future contributors can easily understand the code.
Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes

Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.

What to Expect for the Reviews

The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
After the review, the reviewer will put an action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You

Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

Yard1

Makes sense, thanks

Yard1 · 2024-07-09T23:24:25Z

vllm/engine/async_llm_engine.py

        for request_output in request_outputs:
            self._request_tracker.process_request_output(
                request_output, verbose=self.log_requests)
+            finished = finished or request_output.finished


should this be and, actually? we should stop only if all requests are finished.

Yes. I have rectified it, Thanks.

njhill

Thanks @pushan01! I think these fixes are also needed though

vllm/engine/async_llm_engine.py

In async llm engine, The request will be aborted after finished `(async_llm_engine.py:L135`), but the `engine_step` may still return `True` due to the difference between request finish and engine generate finish judgement. If the engine request is aborted and but the `engine_step` return `True`, the step would be residual till next request arrive, which makes engine die due to timeout error if the next request arrive after ENGINE_ITERATION_TIMEOUT_S. This PR fixed this problem by keep up with the engine_step and the request finish. Signed-off-by: yatta zhang <[email protected]> Signed-off-by: zhangyuntao.dev <[email protected]>

Signed-off-by: yatta zhang <[email protected]> Signed-off-by: zhangyuntao.dev <[email protected]> Co-authored-by: zhangyuntao.dev <[email protected]> (cherry picked from commit 546b101)

Signed-off-by: yatta zhang <[email protected]> Signed-off-by: zhangyuntao.dev <[email protected]> Co-authored-by: zhangyuntao.dev <[email protected]>

Signed-off-by: yatta zhang <[email protected]> Signed-off-by: zhangyuntao.dev <[email protected]> Co-authored-by: zhangyuntao.dev <[email protected]> Signed-off-by: Alvant <[email protected]>

pushan01 changed the title ~~fix(async_engine): fix engine timeout due to request abort~~ [BugFix]: fix engine timeout due to request abort Jul 9, 2024

DarkLight1337 requested a review from robertgshaw2-neuralmagic July 9, 2024 09:39

Yard1 approved these changes Jul 9, 2024

View reviewed changes

Yard1 requested changes Jul 9, 2024

View reviewed changes

pushan01 force-pushed the dev/fix_engine_timeout branch 3 times, most recently from 1fd1598 to cd6179d Compare July 10, 2024 02:49

pushan01 requested a review from Yard1 July 10, 2024 02:55

njhill requested changes Jul 11, 2024

View reviewed changes

vllm/engine/async_llm_engine.py Outdated Show resolved Hide resolved

vllm/engine/async_llm_engine.py Outdated Show resolved Hide resolved

pushan01 force-pushed the dev/fix_engine_timeout branch from cd6179d to 06af52d Compare July 11, 2024 02:39

pushan01 requested a review from njhill July 11, 2024 02:44

Yard1 approved these changes Jul 11, 2024

View reviewed changes

njhill approved these changes Jul 11, 2024

View reviewed changes

njhill merged commit 546b101 into vllm-project:main Jul 11, 2024
71 checks passed

itechbear mentioned this pull request Jul 16, 2024

[Bug]: Engine iteration timed out. This should never happen! #4430

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix]: fix engine timeout due to request abort #6255

[BugFix]: fix engine timeout due to request abort #6255

pushan01 commented Jul 9, 2024 •

edited

Loading

Yard1 left a comment

Yard1 Jul 9, 2024

pushan01 Jul 10, 2024 •

edited

Loading

njhill left a comment

[BugFix]: fix engine timeout due to request abort #6255

[BugFix]: fix engine timeout due to request abort #6255

Conversation

pushan01 commented Jul 9, 2024 • edited Loading

PR Title and Classification

Code Quality

Notes for Large Changes

What to Expect for the Reviews

Thank You

Yard1 left a comment

Choose a reason for hiding this comment

Yard1 Jul 9, 2024

Choose a reason for hiding this comment

pushan01 Jul 10, 2024 • edited Loading

Choose a reason for hiding this comment

njhill left a comment

Choose a reason for hiding this comment

pushan01 commented Jul 9, 2024 •

edited

Loading

pushan01 Jul 10, 2024 •

edited

Loading