[core][distributed] add stateless process group

[youkaichao]

No description provided.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[youkaichao]

merge for rlhf integration

[kwen2501]

LGTM.

General question:
(1) Do you want the new PG to be able to handle object collectives? If so, it seems its method surface would be similar to today PG's method surface. If that's the case, can we (PyTorch) re-work the new_group API so that it does not need calling init_process_group before it? IIUC, init_process_group is the main API that creates a global "state".

(2) If you don't need object collectives, would creating separate "Backend" objects work for you? I guess the "Backend" object would be similar to the PyncclCommunicator here.

[kwen2501]

typo?

[youkaichao]

it's by design, we only use gloo group to initialize this nccl communicator. gloo for control-plane communication, while nccl for data plane.

[youkaichao]

can we (PyTorch) re-work the new_group API so that it does not need calling init_process_group before it?

yes that would be great!

would creating separate "Backend" objects work for you?

not sure how that works. i remember pytorch's nccl backend has some strange operations, like bucketing the transfer, and some logging wrapper for debugging. we don't want these.