[Bug]: The speed of loading the qwen2 72b model, glm-4-9b-chat-1m model in v0.5.0 is much lower than that in v0.4.2. #5523
Comments
|
same issue. The "generating GPU P2P access cache" progress is very slow. |
majestichou
changed the title
|
I also tested the loading of the glm-4-9b-chat-1m model. The speed of loading the glm-4-9b-chat-1m model in v0.5.0 is also much lower than that in v0.4.2. The "generating GPU P2P access cache" progress is very slow. |
|
The first "generating GPU P2P access cache" is very slow which is not related with specify models I think, if the file exists, it will reading from the file directly. |
|
I understand "generating GPU P2P access cache" can be slow, because it needs to test the p2p of each gpu pair. I don't think it can take 20 minutes though. In my experience that's kind of 1~2 minutes. |
can you reproduce my step? And you will see what i said. |
|
Can you try #5528 and give some feedback? I think that should solve this issue. |
What should I try? I use a docker image to start the service instead of compiling the source code. |
@youkaichao Maybe we need a ci to build nightly package for pr? Locally build and install is too boring and slow. This is my test results: v0.5.0: 8min50s pr #5528 (I just remove the vllm config |
|
I try to load qwen2_57b_a14b_instruct and met the same issue. |
|
@NiuBlibing glad it helps, the PR should be merged recently.
It's ongoing. Since most PR only touches Python files, you can also manually replace these files to test. |
Your current environment
I run python3 collect_env.py in the docker container started by vllm:0.5.0, and the output is as follows:
🐛 Describe the bug
I use vLLM (v0.5.0 docker image) to load Qwen/Qwen2-72B-Instruct model. I download the model weights to a local directory named /serving/data/models/. Run the following command to start the Docker container:
docker run --gpus 8 -p 8080:8000 --ipc=host -v /serving/data/models/:/data/models/ --env "TRANSFORMERS_OFFLINE=1" --env "HF_DATASET_OFFLINE=1" vllm/vllm-openai:v0.5.0 --model "/data/models/Qwen2-72B-Instruct/" --tensor-parallel-size 8It took 40 minutes to start the model with vllm version 0.5.0((v0.5.0 docker image). During the boot process, I use nvitop to check the gpu memory usage of the machine. The usage of the display memory stays at 5.4% for a long time. The GPU memory usage stays at 5.4% for a long time, sometimes increases to 5.6%, and then decreases to 5.4%.
Sometimes, the message "zombile process" is displayed on the screen of the nvitop.I used the https://docs.vllm.ai/en/stable/getting_started/debugging.html link to troubleshoot, everthing is ok, I didn't find any strange problems.
The docker log is as follows:
The docker logs are not updated for a long time.
Then I booted the model on the same machine with vllm version 0.4.2.
docker run --gpus 8 -p 8080:8000 --ipc=host -v /serving/data/models/:/data/models/ --env "TRANSFORMERS_OFFLINE=1" --env "HF_DATASET_OFFLINE=1" vllm/vllm-openai:v0.4.2 --model "/data/models/Qwen2-72B-Instruct/" --tensor-parallel-size 8It took 20 minutes to start the model with vllm version 0.4.2((v0.4.2 docker image).
The speed of loading the qwen2 72b model in v0.5.0 is much lower than that in v0.4.2.
I repeated the above process many times, with the same conclusion.
The text was updated successfully, but these errors were encountered: