Refactor CUDA versions in dependencies.yaml.

[bdice]

Description

This is a follow-up PR to #1414. I thought some more about how to separate cuda-version pinnings (which control the CUDA version we use to build and test in conda) from actual CUDA Toolkit package dependencies (which we can handle according to only the major version 11/12). I discussed this PR on a call with @jameslamb in the context of upgrading to CUDA 12.2 (rapidsai/build-planning#6). This set of changes is mostly important for conda builds/tests, since cuda-version only controls conda. The pip wheel build/test process is unchanged, since its CUDA versions are controlled by the shared-workflows CI images.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[bdice]

This dependency list is empty for CUDA 12. I considered listing cuda-cudart-dev here but no packages aside from the compiler are technically needed for CUDA 12. The cuda-nvcc package (from the build dependency list) will include the cuda-cudart pieces needed for runtime. I can't think of a current use of dependencies.yaml where this distinction would matter, but it's possible that we might want to add cuda-cudart-dev here in the future for reasons I can't think of at the moment. I decided to leave it out because it would result in a new line for cuda-cudart-dev in the conda/environments/all_cuda-120_arch-x86_64.yaml file which didn't seem necessary. (Also discussed with @jameslamb)

[jakirkham]

So in cuDF (or another RAPIDS library), this would include a few more packages

[bdice]

Yes, that's correct.

[bdice]

The inclusion of cudatoolkit is now handled by a separate dependency list called cuda which abstracts over major versions instead of major.minor versions. It turns out that the dependency files where the cudatoolkit dependency list was being included should already get their required CUDA parts as dependencies of the built rmm packages (for test_cpp, test_python, or docs environments) or cuda-nvcc itself (if using build.sh with conda/environments/all_cuda-120_arch-x86_64.yaml, see also https://github.com/rapidsai/rmm/pull/1422/files#r1446707857).

Therefore, the only relevant purpose this dependency list must serve is to constrain the CUDA major.minor version by providing a pinning on cuda-version that matches the input matrix's cuda key.

[jakirkham]

Seems like a nice simplification. Thanks Bradley! 🙏

[jameslamb]

This all makes sense to me and should make future upgrades a little easier. Thanks for taking the time to explain it to me!

I'll try something similar with cudf tomorrow.

[jakirkham]

Does rapids-dependency-file-generator have some concept of templating that we could use?

If so, maybe we could rewrite this to something like and allow that to be reused across different CUDA versions and cutdown on the boilerplate further

          - matrix:
              cuda: "{{ cuda }}"
            packages:
              - cuda-version={{ cuda }}

[bdice]

Currently, there is not support for anything like that. I think we’ve avoided introducing a templating layer on purpose, in part. It’s a little bit like how conda recipes want to move away from template logic… we’ve gotten quite far without it for dependencies, but I see the value.

[jakirkham]

For clarity Conda recipes will still have templating. It's more that we want the recipe format to reflect YAML. The reason being we can read them with a normal YAML library in one pass (not lots of passes through with Jinja, selectors, etc.)

There are ways to have templating and make it YAML friendly. Though this is maybe another discussion for a different issue

[jakirkham]

Thanks Bradley! 🙏

[jakirkham]

/merge