[Doc] Fix getting stared to use publicly available model #3963
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previous model `meta/llama-2` required special access to the model card on hugging face leading to an exception when starting the reference example. The new model proposed for the getting started -mistralai/Mistral-7B-Instruct-v0.2- is publicly available and runs on commodity hardware Also update dead link
WoosukKwon
reviewed
Apr 10, 2024
| @@ -4,7 +4,7 @@ vLLM provides an HTTP server that implements OpenAI's [Completions](https://plat | |||
|
|
|||
| You can start the server using Python, or using [Docker](deploying_with_docker.rst): | |||
| ```bash | |||
| python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-hf --dtype float32 --api-key token-abc123 | |||
| python -m vllm.entrypoints.openai.api_server --model mistralai/Mistral-7B-Instruct-v0.2 --dtype auto --api-key token-abc123 | |||
There was a problem hiding this comment.
nit: Let's remove --api-key?
There was a problem hiding this comment.
I suggest to keep the api key arg. Its a low overhead way to ensure basics of access control, especially given the prive of GPU. Having the api key in the QuickStart promotes security best practices (so many quick start configs end up being the production ones..) let's make this one right :)
simon-mo
approved these changes
Apr 10, 2024
simon-mo
reviewed
Apr 10, 2024
| messages=[ | ||
| {"role": "system", "content": "You are a helpful assistant."}, |
SageMoore
pushed a commit
to neuralmagic/nm-vllm
that referenced
this pull request
Apr 11, 2024
andy-neuma
pushed a commit
to neuralmagic/nm-vllm
that referenced
this pull request
Apr 12, 2024
z103cb
pushed a commit
to z103cb/opendatahub_vllm
that referenced
this pull request
Apr 22, 2024
Temirulan
pushed a commit
to Temirulan/vllm-whisper
that referenced
this pull request
Sep 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previous model
meta/llama-2required special access to the model card on hugging face leading to an exception when starting the reference example.The access error with
meta-llama/Llama-2-7b-hfwas:The new model proposed for the getting started -
mistralai/Mistral-7B-Instruct-v0.2- is publicly available and runs on commodity hardware. The getting started proposed in the vLLM Quickstart now works.Also update dead link