vllm cpu docker. Error: TypeError: invalidInputError() missing 1 required positional argument: 'errMsg'

[Starrylun]

参考文章：vLLM Serving with IPEX-LLM on Intel CPU via Docker
参考链接：https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/DockerGuides/vllm_cpu_docker_quickstart.md#vllm-serving-with-ipex-llm-on-intel-cpu-via-docker

服务启动代码：

bash start-vllm-service.sh

start-vllm-service.sh 文件为

#!/bin/bash
model="/home/usr/A000Files/A003Model/qwen/Qwen2-0___5B-Instruct"
served_model_name="qwen"


python -m ipex_llm.vllm.cpu.entrypoints.openai.api_server \
  --served-model-name $served_model_name \
  --port 8899 \
  --model $model \
  --trust-remote-code \
  --device cpu \
  --dtype bfloat16 \
  --enforce-eager \
  --load-in-low-bit bf16 \
  --max-model-len 4096 \
  --max-num-batched-tokens 10240 \
  --max-num-seqs 12 \
  --tensor-parallel-size 1

请求代码：

## openai 调用已经部署的模型 
from openai import OpenAI
client = OpenAI(base_url="http://0.0.0.0:8899/v1", api_key="EMPTY")


completion = client.chat.completions.create(
    model="qwen",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "hi, how are you"},
    ], 
    temperature=0.45,
    max_tokens=2048,
    top_p=0.7,
    n=1,
    frequency_penalty=0.5,
    presence_penalty=0.6,
    stop=None
)
response = completion.choices[0].message.content
response

vllm服务端报错结果展示：

INFO 08-08 05:47:56 async_llm_engine.py:120] Finished request cmpl-d1bf029528104215b052b07957271a95.
INFO:     127.0.0.1:56592 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.11/dist-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.11/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.11/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/ipex_llm/vllm/cpu/entrypoints/openai/api_server.py", line 117, in create_chat_completion
    invalidInputError(isinstance(generator, ChatCompletionResponse))
TypeError: invalidInputError() missing 1 required positional argument: 'errMsg'

openai 请求端报错展示：

File /usr/local/lib/python3.11/dist-packages/openai/_base_client.py:1079, in SyncAPIClient._retry_request(self, options, cast_to, remaining_retries, response_headers, stream, stream_cls)
   1075 # In a synchronous context we are blocking the entire thread. Up to the library user to run the client in a
   1076 # different thread if necessary.
   1077 time.sleep(timeout)
-> 1079 return self._request(
   1080     options=options,
   1081     cast_to=cast_to,
   1082     remaining_retries=remaining,
   1083     stream=stream,
   1084     stream_cls=stream_cls,
   1085 )

File /usr/local/lib/python3.11/dist-packages/openai/_base_client.py:1046, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls)
   1043         err.response.read()
   1045     log.debug("Re-raising status error")
-> 1046     raise self._make_status_error_from_response(err.response) from None
   1048 return self._process_response(
   1049     cast_to=cast_to,
   1050     options=options,
   (...)
   1053     stream_cls=stream_cls,
   1054 )

InternalServerError: Internal Server Error

[Starrylun]

此外还尝试了 request.post 请求，同样也报错了

request.post 请求代码：

%%time
import requests
import json

url = "http://0.0.0.0:8899/v1/chat/completions"  # 注意路径可能需要根据实际API有所不同

headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer sk-no-key-required"  # 你的API密钥
}

data = {
    "model": "qwen",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "hi, how are you"},
    ], 
    'stream': False,
    'temperature': 0.4,
    'repetition_penalty': 1.4,
    'top_p': 0.7,
    'n': 1,
    'max_tokens': 2048, 
}

response = requests.post(url, headers=headers, data=json.dumps(data))

if response.status_code == 200:
    res = response.json()
    print(res['choices'][0]['message']['content'])
else:
    print(f"Request failed with status {response.status_code}: {response.text}")

服务端报错和上面 一样。

---

请求段报错如下：

Request failed with status 500: Internal Server Error
CPU times: user 5.03 ms, sys: 88 μs, total: 5.11 ms
Wall time: 3.26 s

[Starrylun]

另外还直接使用了 文档中提供的 curl 请求，同样也报错了

请求代码：

curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "qwen",
  "prompt": "San Francisco is a",
  "max_tokens": 128,
  "temperature": 0
}' | jq '.choices[0].text'

服务端请求报错类型：

root@sophon-test-001:/llm# curl http://localhost:8899/v1/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "qwen",
  "prompt": "San Francisco is a",
  "max_tokens": 128,
  "temperature": 0
}' | jq '.choices[0].text'
bash: jq: command not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   994  100   898  100    96     98     10  0:00:09  0:00:09 --:--:--   230
curl: (23) Failed writing body

[Starrylun]

设备类型：

root@sophon-test-001:/llm# lscpu 
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 57 bits virtual
  Byte Order:            Little Endian
CPU(s):                  48
  On-line CPU(s) list:   0-47
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz
    CPU family:          6
    Model:               106
    Thread(s) per core:  2
    Core(s) per socket:  12
    Socket(s):           2
    Stepping:            6
    CPU max MHz:         3300.0000
    CPU min MHz:         800.0000
    BogoMIPS:            4200.00
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mm
                         x fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon peb
                         s bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_
                         cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcn
                         t tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ca
                         t_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexp
                         riority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx5
                         12f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw a
                         vx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local sp
                         lit_lock_detect wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx
                         512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx5
                         12_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities

[xiangyuT]

Hi @Starrylun,
I have reproduced the issue, and it appears to be a bug in the /v1/chat/completions endpoint within ipex_llm.vllm.cpu.entrypoints.openai.apiserver. This should be fixed by #11748. You can try the next nightly-build Docker image (with ipex-llm version >= 2.1.0b20240809) and I will update this thread once the new Docker image is released.

root@sophon-test-001:/llm# curl http://localhost:8899/v1/completions
-H "Content-Type: application/json"
-d '{
"model": "qwen",
"prompt": "San Francisco is a",
"max_tokens": 128,
"temperature": 0
}' | jq '.choices[0].text'
bash: jq: command not found

BTW, the error message here indicates jq is not installed in your environment. You can fix this by running apt install jq.

[xiangyuT]

Hi @Starrylun,
The new Docker image has been released. Please update to the latest version and try again. It works fine in my environment.

[Starrylun]

Hi @Starrylun, The new Docker image has been released. Please update to the latest version and try again. It works fine in my environment.

thanks, update the latest images, it works fine.

[KiwiHana]

Solved. Please close this issue.