Setup CI job to automate testing with RAPIDS

[jrhemstad]

Tasks

Beta Give feedback

1. Add jobs for RMM
2. Add jobs for cuDF

[bdice]

@jrhemstad asked me to share some thoughts on implementation.

Which RAPIDS branch should we use?

The decision is probably between main (the last release) and the default branch (the current development branch, which is branch-24.06 at present). The tradeoff here is that using main will probably be more stable and would help answer the question "did recent changes in CCCL break RAPIDS?" However, the goal of testing here is really to build more visibility into the partnership between CCCL and RAPIDS. We expect breakage will continue to occur in both CCCL and RAPIDS, because both libraries are evolving at a fast pace. For this reason, I think using the default branch is the better choice. This way, breaking changes in CCCL can co-evolve with changes in RAPIDS that are meant to work around any problem cases found in CCCL (hopefully we can make fixes to RAPIDS that make it compatible with the latest CCCL as well as the pinned version in rapids-cmake).

How should we test this?

We can probably get by with just one build configuration for now. CUDA 12, gcc 11. We can use the Docker container rapidsai/ci:latest for this, or maybe a devcontainer CI image? To force RMM/cuDF to build with the latest CCCL, we'd need to override the pinnings from rapids-cmake's versions.json and also manage a set of patches that RAPIDS has to apply on top of CCCL. Perhaps @robertmaynard has opinions on the best way to do this.

Maybe we want to have a layout like:

rmm-integration

• build.sh -- script to clone, patch, and build RMM (ideally this calls RMM's own build.sh)
• patches/ -- directory of CCCL patches