[TASK][MEDIUM] Integrate NVIDIA RAPIDS with Spark engine

[pan3793]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before creating

• I have searched in the task list and found no similar tasks.

Mentor

• I have sufficient knowledge and experience of this task, and I volunteer to be the mentor of this task to guide contributors to complete the task.

Skill requirements

• Basic knowledge of Scala programming language, Kyuubi architecture, Apache Spark, NVIDIA RAPIDS
• Basic English writing skills

Background and Goals

RAPIDS is a Spark plugin developed by NVIDIA to accelerate Apache Spark with GPUs.

As Kyuubi currently supports using Spark as the computing engine, we can add tests to integrate with RAPIDS in the Spark engine, and write docs to guide users in setting up Kyuubi with Spark plus RAPIDS to leverage GPUs.

Implementation steps

• Add RAPIDS UTs in spark engine module, which could be conditionally enabled via maven profile and scalatest annotation
• Setup CI for RAPIDS testing (I'm not sure if GitHub Actions or ASF infra could provide such CUDA environments for running CI)
• Write docs to guide users in setting up Kyuubi with Spark plus RAPIDS to leverage GPUs.

Additional context

Introduction of #6232

[tangzhao132]

Hi I would like to try this task

[pan3793]

@tangzhao132 do you have an estimated time to complete this task? see additional context #5357

[tangzhao132]

@tangzhao132 do you have an estimated time to complete this task? see additional context #5357

I think this task is challenging for a newbee, so I think I could finish before Nov 18

[pan3793]

@tangzhao132 thanks for your feedback, please let me know if you encounter any issues during the integration, looking forward to your contributions~

BTW, if you find any issues during the integration, feel free to raise PRs to fix/improve it, small and independent patches are always welcome~