[Core][Distributed] fix pynccl del error

[youkaichao]

It is observed from #4488 , that CI actually has errors, although it is ignored. Therefore, essentially the ncclCommDestroy in NCCLCommunicator.__del__ is never called. We can remove the code to avoid the CI error bothering users.

[youkaichao]

TL;DR; destruction of nccl is a collective call, like a broadcast or allreduce, and thus is blocking. However, it is not guaranteed to be called in the same order because Python's garbage collection system destructs objects in random order.

We cannot rely on Python's garbage collection system to work here, because The destruction of modules and objects in modules is done in random order.

The driver process holds a communicator and a handle to ray actor, the worker process holds a communicator.

If the driver process calls del for the communicator first, it will stuck, because it is a collective call, and it need to wait for the other process to delete the communicator as well. However, that worker process will run the main loop to wait for command from the driver process. Thus, we will see a deadlock.

If the driver process calls del for the ray actor first, the worker process will hang in the collective call, until the driver process also called the collective call to destroy the communicator.

Things can go crazy when we have multiple communicators (e.g. PyTorch nccl backend has nccl communicators), because it will require the same destruction order for these communicators, which is essentially impossible for Python.

One possible solution is to add cleanup logic in ray_gpu_executor's __del__ function. That's also questionable, because ray module might also be destroyed before ray_gpu_executor.

The ultimate solution might be to provide some context manager like with vllm.context(), and to ask users place any vllm specific code under that context. This way, we can do something under __exit__ , before the Python's garbage collection system starts to shutdown the Python interpreter.

[youkaichao]

Per our offline discussion with @zhuohan123 @WoosukKwon @simon-mo @LiuXiaoxuanPKU , we can just skip the destruction to avoid deadlocks.