[Core] remove cupy dependency

[youkaichao]

Separate code from #3442 , only remove cupy dependency.

[youkaichao]

Support evidence from #3617 : this PR solved cupy initialization issue 👍

[WoosukKwon]

Can we get confirmation from AMD people that we can do the same thing to replace PyTorch RCCL? While I believe we don't have to do this in the current PR, I'd like to get confirmation and find someone who takes that.

[simon-mo]

i made an initial pass and it looks good overall. I think we are maybe lacking a bit more tests in the test_pynccl.py?

[youkaichao]

@WoosukKwon @simon-mo thanks for the review! I talked to amd folks to ask for review. Let's see if we can get feedback today. If not, we can merge first, and let them send fix PRs later.

[WoosukKwon]

Thanks for the great work! Left some questions.

[WoosukKwon]

I think this part can be deleted as we remove CuPy?

[youkaichao]

I don't know why this code was needed before. So I don't want to change it 🤣 If you have more context, we can discuss if we can delete this.

[WoosukKwon]

When using CuPy + CUDA graphs, deadlocks happen when the CuPy backend is deleted before the CUDA graphs using it are deleted. I actually don't know the reason for this, but this doesn't happen when using NCCL through PyTorch probably because the NCCL communicator managed by PyTorch is deleted at the very end.

[WoosukKwon]

@youkaichao could you check whether we can delete this? You can simply run python examples/llm_engine_example.py -tp 2 and see if the process hangs.

[youkaichao]

Okay, will have a try.

[WoosukKwon]

BTW, I do agree with you that we can remove this code some time later. If you'd like to do so, could you please add TODO in the code?

[youkaichao]

I tested the code (removing the above two lines) for 100 times for i in {1..100}; do python3 examples/llm_engine_example.py -tp=2 && echo "Test passed $i times" || break; done , and I don't see any deadlocks. This gives us confidence in removing the code later. I left a comment there to remove it in v0.4.1 .

[youkaichao]

Overall I think we should take a small step for every release. Distributed related bugs like hang/deadlock are highly unstable and difficult to test.

My plan is to use pynccl as an equivalent replacement of cupy in v0.4.0 , and collect user feedback to decide the next move for v0.4.1 (e.g. removing tricky code wrote for cupy).

cc @WoosukKwon

[WoosukKwon]

Thanks again for the great work!

[jojac47]

Hello everyone,
I was able to get this resolved myself as well without removing cuppy as a dependency. What I had to do was Instead of using the VLLM or Nvidia base containers, I used the official Cupy(which includes the dependencies for the Nvidia drivers) image as my new base image and followed the Linux install instructions inside the container and committed it to a new image.