rearrange dependencies.yaml, fix development instructions

[jameslamb]

Description

Follow-up to #1448. This proposes some dependencies.yaml changes that I noticed while working on adding libcuspatial wheels in #1450.

I've left inline comments with more details, but in short:

• not including any cuspatial projects in the conda environment files checked into this repo (which are intended to be used to set up a development environment)
• removing an unnecessary conda-only list of dependencies from pyproject.toml-specific config for cuproj
• introduction of depends_on_libcuspatial and depends_on_cuproj lists in dependencies.yaml, to reduce duplication

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[jameslamb]

See the build instructions in the development docs:

cuspatial/docs/source/developer_guide/build.md

Lines 15 to 32 in 1544e7b

	## Install dependencies
	1. `export CUSPATIAL_HOME=$(pwd)/cuspatial`
	2. clone the cuSpatial repo
	```shell
	conda env create -n cuspatial --file conda/environments/all_cuda-118_arch-x86_64.yaml
	```
	## Build cuSpatial
	### From the cuSpatial Dev Container:
	Execute `build-cuspatial-cpp` to build `libcuspatial`. The following options may be added.
	- `-DBUILD_TESTS=ON`: build `libcuspatial` unit tests.
	- `-DBUILD_BENCHMARKS=ON`: build `libcuspatial` benchmarks.
	- `-DCMAKE_BUILD_TYPE=Debug`: Create a Debug build of `libcuspatial` (default is Release).
	In addition, `build-cuspatial-python` to build cuspatial cython components.

1. they recommend starting by creating a conda environment from one of the files in conda/environments
2. they recommend building libcuspatial and its tests from source

Those steps are currently broken.

The libcuspatial-tests conda package is not distributed anywhere, so that env creation will fail like this:

Solving environment: failed

ResolvePackageNotFound: 
  - libcuspatial-tests[version='24.10.*,>=0.0.0a0']

Here, I'm recommending dropping those packages from these files. Similar to how for example you won't find e.g. libcuml in these files in the cuml repo (code link) or libcuml in the cudf repo (code link).

[jameslamb]

This section populates the [test] extra for cuproj wheels.

The test_cuspatial list contained only conda dependencies, so it didn't affect wheels in any way. That's why you see this being removed without replacement.

[jameslamb]

#1450 will add requirements / pyproject sections to this. They don't need to exist until libcuspatial wheels exist.

[jameslamb]

Notice that here, I've taken a list that previously contained cuproj, cuspatial, and libcuspatial with only cuproj and cuspatial.

That's intentional.... this test_python list is used for testing the Python conda packages:

cuspatial/ci/test_python.sh

Lines 13 to 17 in 1544e7b

	rapids-dependency-file-generator \
	--output conda \
	--file-key test_python \
	--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" \
	--prepend-channel "${CPP_CHANNEL}" --prepend-channel "${PYTHON_CHANNEL}" | tee env.yaml

Omitting libcuspatial here gives us a chance to learn about packaging errors of the form "cuspatial is missing a dependency on `libcuspatial" in CI.

[harrism]

Are you planning on fixing the instructions, as the title of this PR says?

[jameslamb]

Are you planning on fixing the instructions, as the title of this PR says

The changes to dependencies.yaml do fix the instructions... now following this step in the docs will succeed:

cuspatial/docs/source/developer_guide/build.md

Line 21 in 0e9d36c

conda env create -n cuspatial --file conda/environments/all_cuda-118_arch-x86_64.yaml

where previously that would have failed because libcuspatial-tests is not distributed anywhere.

Sorry if the wording in the title here was unclear, I can see how it makes it seems like this might modify those instructions.

[bdice]

Approving with a question about best practices… which I partly answered for myself. Happy to hear feedback but I think no changes are required.

[bdice]

I’m not sure if I like this structure where we have a dependency list that defines libcuspatial-tests. In libcudf tests we list the libcudf-tests package explicitly rather than using a dependency list. https://github.com/rapidsai/cudf/blob/7018a33be752da9363db5431560d8d12bf378920/ci/test_cpp_common.sh#L34

The *-tests conda packages are kind of special in that they are only CI artifacts and thus never available from any “proper” conda channels, only local channels of CI built artifacts. For that reason I am hesitant to use RDFG to define them

[bdice]

Hmmmm. The alternate proposal I had in mind (removing the test_libcuspatial dependency list and installing libcuspatial-tests manually) would not make sense for this repo, since we’ve moved to the “single step solve” for C++ tests in cuspatial.

cuspatial/ci/test_cpp.sh

Line 17 in 0e9d36c

--prepend-channel "${CPP_CHANNEL}" | tee env.yaml

Maybe we do have the right approach and no changes are needed…

[jameslamb]

yep exactly, the use of that --prepend-channel pattern here is why I left this as-is.

[jameslamb]

/merge