[Request] Support to use ray as the distributed inference

[nkwangleiGIT]

🥰 需求描述 | Feature Description

需要支持使用 Ray 作为分布式推理集群

🧐 解决方案 | Proposed Solution

模型服务部署时，需要支持以下配置（高级配置中）：
1、选择 Ray 集群进行分布式推理
参考：http://kubeagi.k8s.com.cn/docs/Configuration/DistributedInference/run-inference-using-ray

    rayClusters:
    - name: 3090-2-GPUs
      headAddress: raycluster-kuberay-head-svc.kuberay-system.svc:6379
      pythonVersion: 3.9.18

页面上支持选择使用哪个集群，默认选择第一个

2. 配置额外的环境变量

  # 配置需要的 GPU 数量
  - name: NUMBER_GPUS
    value: "2"
  # 选择的 Ray 集群的索引号
  - name: RAY_CLUSTER_INDEX
    value: "0"
  # the GPU IDs will be ordered by pci bus IDs.
  - name: CUDA_DEVICE_ORDER
    value: "PCI_BUS_ID"
  # specify which GPU(s) to be used
  - name: CUDA_VISIBLE_DEVICES
    value: "0,3"

📝 补充信息 | Additional Information

No response

[yunti-bot]

👀 @nkwangleiGIT
Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.
非常感谢您提交 issue。我们会尽快调查此事，并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

[Carrotzpc]

@shengkan9 支持下这块儿，主要是模型服务部署的高级配置需要新增 Ray 的支持

[nkwangleiGIT]

I think it's done

[yunti-bot]

✅ @nkwangleiGIT


This issue is closed, If you have any questions, you can comment and reply.
此问题已经关闭。如果您有任何问题，可以留言并回复。