[Bug]: Invalid inputs do not result in error

[dyastremsky]

Your current environment

Start vLLM: docker run --gpus all -v ~/.cache/huggingface:/root/.cache/huggingface -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model gpt2 --dtype float16 --max-model-len 1024

🐛 Describe the bug

When running inference with vLLM, errors are not returned when an invalid input is supplied.

After starting vLLM as specified above, run a curl command that supplies invalid inputs: curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "gpt2", "messages": [ { "role": "system", "content": "You are a helpful assistant.", "fake_input": "2" }, { "role": "user", "content": "Hello!", "fake_input": "3" } ] }'

You get a successful response. Fake_input is dropped instead of returning an error. This silent failure can be dangerous with typos or invalid inputs. Can this be fixed?

[simon-mo]

Thanks for raising this issue. It does puzzle me why it happen since we have the pydantic model defined here

vllm/vllm/entrypoints/openai/protocol.py

Line 63 in aae0824

class ChatCompletionRequest(BaseModel):

and fastapi reference here

vllm/vllm/entrypoints/openai/api_server.py

Line 88 in aae0824

async def create_chat_completion(request: ChatCompletionRequest,

We also did not enable extra fields allowed.

[simon-mo]

Any help appreciated!

[DarkLight1337]

The default setting in Pydantic (as documented here) silently drops extra fields.

In order to disallow extra fields, we have to explicitly disable it in the model class (extra="forbid"). Going to open a PR shortly.

[DarkLight1337]

I have tested this and found that there is another reason why invalid keys in messages isn't being checked. Currently, messages is being annotated as Dict[str, str] so the schema is not fully specified. PR #3467 improves the type annotations but still does not limit the allowed keys. To fix this issue, I'm going to port over some type annotations from my WIP PR #4200.

Edit: Opened PR #4355.

[dyastremsky]

Wow, that was lightning fast, @DarkLight1337 and @simon-mo. Thanks for working on this so quickly!