[Performance]: The impact of CPU on vLLM performance is significant.

[skylee-01]

Proposal to improve performance

We used the same GPU on two machines but different CPUs. The following experimental conclusions were drawn:
Experimental results: The GPU is 3090, and the CPU was upgraded from Xeon Gold 6240 to i9-12900k. The impact is as follows.
a. vLLM achieved a 3.8x speedup in the agent scenario.
b. TGi achieved a 1.23x speedup in the agent scenario.
c. vLLM still has latency issues, but the time has been reduced to 100ms (previously 300ms).
e. GPU utilization has increased from 70% to 90%.

From the stress test data, it is evident that vLLM heavily relies on the performance of the CPU.
What are the main factors affecting CPU performance, and how can they be optimized?

[skylee-01]

Related experiments： #7540

[skylee-01]

@WoosukKwon @youkaichao Please provide some assistance.

[youkaichao]

what is the vllm version you use?

[skylee-01]

what is the vllm version you use?

0.5.5

[youkaichao]

we are optimizing the cpu time, please stay tuned. it should not be so dependent on CPU performance in the future.

[skylee-01]

we are optimizing the cpu time, please stay tuned. it should not be so dependent on CPU performance in the future.

What is the reason for VLLM's current heavy dependence on CPU, and what are the directions for optimization?
Our team is also trying to participate in the work of VLLM, hoping to contribute to the VLLM community. We hope to be able to submit code for VLLM.

[youkaichao]

cpu needs to serve http requests, and also prepare lots of input data for the GPU, which changes for every step (because of continuous batching and auto-regressive LLM decoding).

for some examples on this line of optimization, see #7000 and #8092

contributions are definitely welcome!

[skylee-01]

cpu needs to serve http requests, and also prepare lots of input data for the GPU, which changes for every step (because of continuous batching and auto-regressive LLM decoding).

for some examples on this line of optimization, see #7000 and #8092

contributions are definitely welcome!

Our team has developed some spec decoding features based on VLLM, which have been used internally and have yielded good performance benefits. How can we join the VLLM project, and where would be a good place to start?

[youkaichao]

welcome to send emails to [email protected]

[robertgshaw2-neuralmagic]

Really interesting. Thanks for reporting. The GPUs are getting fast :)

[WoosukKwon]

Hi @skylee-01 Thanks for reporting this! We also recently discovered the same problem. We plan to do more optimizations to mitigate the CPU effect.

vLLM is a fully open community-driven project, so we'd appreciate any contributions, including submitting or reviewing PRs, answering questions, and helping documentation.