[CI/Build] Avoid downloading all HF files in RemoteOpenAIServer
#7836
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge). To run full CI, you can do one of these:
🚀 |
tests/utils.py
Outdated
| self.host = str(args.host or 'localhost') | ||
| self.port = int(args.port) | ||
|
|
||
| # download the model before starting the server to avoid timeout | ||
| engine_args = AsyncEngineArgs.from_cli_args(args) | ||
| engine_config = engine_args.create_engine_config() |
There was a problem hiding this comment.
can we directly create a load config object? it should be simple in my opinion. just load format auto.
There was a problem hiding this comment.
Let's fix the tests first before simplifying this.
There was a problem hiding this comment.
I'm not that familiar with the model loading code so may need some help regarding fixing the tests.
tests/utils.py
Outdated
| @@ -60,36 +61,40 @@ class RemoteOpenAIServer: | |||
|
|
|||
| def __init__(self, | |||
| model: str, | |||
| cli_args: List[str], | |||
| serve_args: List[str], | |||
There was a problem hiding this comment.
why rename the arg name btw?
There was a problem hiding this comment.
I meant to associate them with vllm serve specifically, not just any CLI. Perhaps it could be clearer.
Edit: Updated the name to vllm_serve_args
|
|
||
| parser = FlexibleArgumentParser( | ||
| description="vLLM's remote OpenAI server.") | ||
| parser = make_arg_parser(parser) | ||
| args = parser.parse_args(cli_args) |
There was a problem hiding this comment.
I think you just need args = parser.parse_args(["--model", model, *cli_args])
DarkLight1337
changed the title
RemoteOpenAIServer
github-actions
bot
added
the
ready
…lm-project#7836) Signed-off-by: Alvant <[email protected]>
Fixes #7834 (comment)