[mypy] Enable following imports for entrypoints

[DarkLight1337]

This PR enables follow-imports=silent for vllm.entrypoints. It also cleans up some of the type annotations related to tokenizer (Union[PreTrainedTokenizer, PreTrainedTokenizerFast] is now replaced with AnyTokenizer).

Partially addresses #3680.

[DarkLight1337]

@rkooo567 please review when you have time.

[DarkLight1337]

@petersalas can you help take a look why the audio entrypoints test now fails to pass pydantic validation?

[petersalas]

@DarkLight1337 can you elaborate a bit? I pulled the branch down but the audio tests seem to pass. Is there a call stack with the failure I can look at?

[rkooo567]

very cool! have the first iteration

[rkooo567]

can you explain to me why we are doing this here instead of cast?

[DarkLight1337]

Imo this way is a bit more readable and also provides additional validation.

[rkooo567]

what is it exactly for? (also doesn't thits have duplicated logics with post_init? should we remove them?)

[DarkLight1337]

It is so that I can pass None as the arguments to SamplingParams without having to go to this file and lookup the default values.

[DarkLight1337]

@DarkLight1337 can you elaborate a bit? I pulled the branch down but the audio tests seem to pass. Is there a call stack with the failure I can look at?

____________________________ test_single_chat_session_audio[https://upload.wikimedia.org/wikipedia/en/b/bf/Dave_Niehaus_Winning_Call_1995_AL_Division_Series.ogg-facebook/opt-125m] _____________________________

client = <openai.AsyncOpenAI object at 0x7f4d1a7715a0>, model_name = 'facebook/opt-125m', audio_url = 'https://upload.wikimedia.org/wikipedia/en/b/bf/Dave_Niehaus_Winning_Call_1995_AL_Division_Series.ogg'

    @pytest.mark.asyncio
    @pytest.mark.parametrize("model_name", [MODEL_NAME])
    @pytest.mark.parametrize("audio_url", TEST_AUDIO_URLS)
    async def test_single_chat_session_audio(client: openai.AsyncOpenAI,
                                             model_name: str, audio_url: str):
        messages = [{
            "role":
            "user",
            "content": [
                {
                    "type": "audio_url",
                    "audio_url": {
                        "url": audio_url
                    }
                },
                {
                    "type": "text",
                    "text": "What's happening in this audio?"
                },
            ],
        }]
    
        # test single completion
>       chat_completion = await client.chat.completions.create(model=model_name,
                                                               messages=messages,
                                                               max_tokens=10,
                                                               logprobs=True,
                                                               top_logprobs=5)

tests/entrypoints/openai/test_audio.py:169: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../miniconda3/envs/vllm/lib/python3.10/site-packages/openai/resources/chat/completions.py:1339: in create
    return await self._post(
../miniconda3/envs/vllm/lib/python3.10/site-packages/openai/_base_client.py:1816: in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
../miniconda3/envs/vllm/lib/python3.10/site-packages/openai/_base_client.py:1510: in request
    return await self._request(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <openai.AsyncOpenAI object at 0x7f4d1a7715a0>, cast_to = <class 'openai.types.chat.chat_completion.ChatCompletion'>
options = FinalRequestOptions(method='post', url='/chat/completions', params={}, headers=NOT_GIVEN, max_retries=NOT_GIVEN, timeo...his audio?"}]}], 'model': 'facebook/opt-125m', 'logprobs': True, 'max_tokens': 10, 'top_logprobs': 5}, extra_json=None)

    async def _request(
        self,
        cast_to: Type[ResponseT],
        options: FinalRequestOptions,
        *,
        stream: bool,
        stream_cls: type[_AsyncStreamT] | None,
        remaining_retries: int | None,
    ) -> ResponseT | _AsyncStreamT:
        if self._platform is None:
            # `get_platform` can make blocking IO calls so we
            # execute it earlier while we are in an async context
            self._platform = await asyncify(get_platform)()
    
        # create a copy of the options we were given so that if the
        # options are mutated later & we then retry, the retries are
        # given the original options
        input_options = model_copy(options)
    
        cast_to = self._maybe_override_cast_to(cast_to, options)
        options = await self._prepare_options(options)
    
        retries = self._remaining_retries(remaining_retries, options)
        request = self._build_request(options)
        await self._prepare_request(request)
    
        kwargs: HttpxSendArgs = {}
        if self.custom_auth is not None:
            kwargs["auth"] = self.custom_auth
    
        try:
            response = await self._client.send(
                request,
                stream=stream or self._should_stream_response_body(request=request),
                **kwargs,
            )
        except httpx.TimeoutException as err:
            log.debug("Encountered httpx.TimeoutException", exc_info=True)
    
            if retries > 0:
                return await self._retry_request(
                    input_options,
                    cast_to,
                    retries,
                    stream=stream,
                    stream_cls=stream_cls,
                    response_headers=None,
                )
    
            log.debug("Raising timeout error")
            raise APITimeoutError(request=request) from err
        except Exception as err:
            log.debug("Encountered Exception", exc_info=True)
    
            if retries > 0:
                return await self._retry_request(
                    input_options,
                    cast_to,
                    retries,
                    stream=stream,
                    stream_cls=stream_cls,
                    response_headers=None,
                )
    
            log.debug("Raising connection error")
            raise APIConnectionError(request=request) from err
    
        log.debug(
            'HTTP Request: %s %s "%i %s"', request.method, request.url, response.status_code, response.reason_phrase
        )
    
        try:
            response.raise_for_status()
        except httpx.HTTPStatusError as err:  # thrown on 4xx and 5xx status code
            log.debug("Encountered httpx.HTTPStatusError", exc_info=True)
    
            if retries > 0 and self._should_retry(err.response):
                await err.response.aclose()
                return await self._retry_request(
                    input_options,
                    cast_to,
                    retries,
                    err.response.headers,
                    stream=stream,
                    stream_cls=stream_cls,
                )
    
            # If the response is streamed then we need to explicitly read the response
            # to completion before attempting to access the response text.
            if not err.response.is_closed:
                await err.response.aread()
    
            log.debug("Re-raising status error")
>           raise self._make_status_error_from_response(err.response) from None
E           openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "6 validation errors for ValidatorIterator\n0.typed-dict.text\n  Field required [type=missing, input_value={'type': 'audio_url', 'au...L_Division_Series.ogg'}}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.7/v/missing\n0.typed-dict.type\n  Input should be 'text' [type=literal_error, input_value='audio_url', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.7/v/literal_error\n0.typed-dict.audio_url\n  Extra inputs are not permitted [type=extra_forbidden, input_value={'url': 'https://upload.w...AL_Division_Series.ogg'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.7/v/extra_forbidden\n0.typed-dict.image_url\n  Field required [type=missing, input_value={'type': 'audio_url', 'au...L_Division_Series.ogg'}}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.7/v/missing\n0.typed-dict.type\n  Input should be 'image_url' [type=literal_error, input_value='audio_url', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.7/v/literal_error\n0.typed-dict.audio_url\n  Extra inputs are not permitted [type=extra_forbidden, input_value={'url': 'https://upload.w...AL_Division_Series.ogg'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.7/v/extra_forbidden", 'type': 'BadRequestError', 'param': None, 'code': 400}

../miniconda3/envs/vllm/lib/python3.10/site-packages/openai/_base_client.py:1611: BadRequestError

The error is thrown server-side while iterating through each part in a message: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/chat_utils.py#L164

I'm using pydantic 2.7.1 and pydantic_core 2.18.2, if that helps. Odd that it's not happening in CI now.

[DarkLight1337]

@DarkLight1337 can you elaborate a bit? I pulled the branch down but the audio tests seem to pass. Is there a call stack with the failure I can look at?

The error is thrown server-side while iterating through each part in a message: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/chat_utils.py#L164

I'm using pydantic 2.7.1 and pydantic_core 2.18.2, if that helps. Odd that it's not happening in CI now.

Maybe this is a bug in Pydantic. I updated to pydantic 2.8.2 and pydantic-core 2.20.1 and don't get this error anymore. Sorry for bothering you!

[rkooo567]

lgtm!