[Feature] Investigate integration of KubeRay and Volcano, add to docs

[DmitriGekhtman]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Investigate use of Volcano with KubeRay for gang scheduling.
Write up a user guide.

cc @kevin85421 @sihanwang41

Use case

Gang scheduling of KubeRay resources.

Related issues

This is a more specific version of #213

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[Jeffwan]

Is this one same as #213?

[DmitriGekhtman]

Re-ordered the text of the issue description to make it clearer:
I think it would be interesting specifically to target Volcano, since it's a popular tool.

Hopefully, KubeRay already exposes the necessary configuration and no changes are required.

Opened this issue on the basis of some discussion with @kevin85421.

[tgaddair]

Can look at how the Spark Operator handles integration as a reference:

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/pkg/batchscheduler/volcano/volcano_scheduler.go

It's not yet clear to me if this requires modifying KubeRay to work, or if native integration just saves some boilerplate.

[DmitriGekhtman]

Well-documented boilerplate is the preferable route when possible.

IMO, KubeRay shouldn't be too opinionated about external tools.

[tgaddair]

From what I can gather, the only real requirement is to create a PodGroup for the Ray Cluster and tie it to the RayCluster CR via the ownerReferences metadata. Assuming this doesn't require some knowledge of RayClusters on the Volcano side, I think it should be doable to just have users create PodGroups manually.

[tgaddair]

The only other element to this is handling updates, like how the autoscaler can adjust the number of replicas. This could also benefit from native integration in KubeRay, as otherwise it could be difficult to intercept these changes to update the PodGroup. But that could be a "buyer beware" thing to put into the docs.