cupy_backends.cuda.libs.nccl.NcclError: NCCL_ERROR_INTERNAL_ERROR: internal error

[QianqianNie]

Hi Im getting the following error with vllm 0.3.2 on A100

engine = AsyncLLMEngine.from_engine_args(engine_args)
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 625, in from_engine_args
engine = cls(parallel_config.worker_use_ray,
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 321, in init
self.engine = self._init_engine(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 366, in _init_engine
return engine_class(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 118, in init
self._init_workers_ray(placement_group)
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 286, in _init_workers_ray
self._run_workers("init_model", cupy_port=get_open_port())
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 1014, in _run_workers
driver_worker_output = getattr(self.driver_worker,
File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 94, in init_model
init_distributed_environment(self.parallel_config, self.rank,
File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 275, in init_distributed_environment
cupy_utils.init_process_group(
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/parallel_utils/cupy_utils.py", line 90, in init_process_group
_NCCL_BACKEND = NCCLBackendWithBFloat16(world_size, rank, host, port)
File "/usr/local/lib/python3.10/dist-packages/cupyx/distributed/_nccl_comm.py", line 70, in init
self._init_with_tcp_store(n_devices, rank, host, port)
File "/usr/local/lib/python3.10/dist-packages/cupyx/distributed/_nccl_comm.py", line 100, in _init_with_tcp_store
self._comm = nccl.NcclCommunicator(n_devices, nccl_id, rank)
File "cupy_backends/cuda/libs/nccl.pyx", line 283, in cupy_backends.cuda.libs.nccl.NcclCommunicator.init
File "cupy_backends/cuda/libs/nccl.pyx", line 129, in cupy_backends.cuda.libs.nccl.check_status
cupy_backends.cuda.libs.nccl.NcclError: NCCL_ERROR_INTERNAL_ERROR: internal error

This is the driver information:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05 Driver Version: 535.154.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A100 80GB PCIe Off | 00000001:00:00.0 Off | 0 |
| N/A 34C P0 51W / 300W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A100 80GB PCIe Off | 00000002:00:00.0 Off | 0 |
| N/A 35C P0 53W / 300W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA A100 80GB PCIe Off | 00000003:00:00.0 Off | 0 |
| N/A 38C P0 54W / 300W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA A100 80GB PCIe Off | 00000004:00:00.0 Off | 0 |
| N/A 37C P0 54W / 300W | 0MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

[olaf-beh]

Might be because of this:

As of now, vLLM’s binaries are compiled with CUDA 12.1

https://docs.vllm.ai/en/latest/getting_started/installation.html#install-with-pip

On debian (guess same on ubuntu) to check if cuda 12.1 is installed do a

ls -ld /usr/local/cuda*

To clarify, above is for cuda-toolkit versions. I think a newer cuda driver e.g. 12.2 is downwards compatible with older cuda-toolkits like 12.1.

[youkaichao]

cupy is removed in #3625 . Please try the new release v0.4.0.post1 .