add PR CI for cugraph-pyg and cugraph-dgl #59

jameslamb · 2024-10-21T16:02:13Z

Another steps towards completing the work started in #53

Fixes #15

Contributes to rapidsai/build-planning#111

Proposes changes to get CI running on pull requests for cugraph-pyg and cugraph-dgl

Notes for Reviewers

Workflows for nightly builds and publishing nightly packages are intentionally not included here. See #58 (comment)

Notebook tests are intentionally not added here... they'll be added in the next PR.

Pulls in changes from these other upstream PRs that had not been ported over to this repo:

…ey should express them

…s again

ci/test_wheel_cugraph-dgl.sh

jameslamb · 2024-10-28T21:33:02Z

🥳 this is now ready for review! Thanks very much for all the help @alexbarghi-nv .

… into add-pyg-and-dgl-ci

alexbarghi-nv

Just looked this over one last time; I think this is ready to be merged. Thanks for all your hard work @jameslamb

bdice · 2024-10-29T19:29:19Z

.gitignore

+dataset/
+datasets/


Do we really need both of these paths? Why?

Good point, there's an opportunity to simplify here.

I added dataset/ because I saw files created at python/cugraph-pyg/cugraph_pyg/dataset/ when running the cugraph-pyg examples locally.

Looks like that comes from here: https://github.com/snap-stanford/ogb/blob/f631af76359c9687b2fe60905557bbb241916258/ogb/nodeproppred/dataset.py#L13

I just pushed changes writing all those datasets to datasets/ instead, so we don't need to have both rules in gitignore.

bdice · 2024-10-29T19:30:40Z

ci/build_python.sh

+RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild \
+  --no-test \
+  --channel "${CPP_CHANNEL}" \
+  --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \


Do we need ${RAPIDS_CONDA_BLD_OUTPUT_DIR} here? What packages is it supplying? I don't see pylibwholegraph in the dependencies of cugraph-pyg or cugraph-dgl.

Same for cugraph-dgl below.

You're totally right. pylibwholegraph is a testing dependency of both cugraph-pyg and cugraph-dgl, but not a build-time dependency, so --channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" is unnecessary. I just pushed 1202c36 removing it.

bdice · 2024-10-29T19:36:19Z

ci/test_wheel_cugraph-pyg.sh

+# Used to skip certain examples in CI due to memory limitations
+export CI_RUN=1


Typically the environment variable CI is set to true in CI contexts. https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/store-information-in-variables#default-environment-variables

We could use that to avoid introducing a separate variable.

Yeah great point! Looks to me like this was carried over from cugraph, but there it's only used for the narrow purpose of cugraph-pyg, so we can just simplify it here.

Did that in 1202c36

bdice · 2024-10-29T19:47:59Z

It seems like there are opportunities to move from CUDA 12.1 to 12.4 and drop the pytorch channel (in favor of conda-forge) in future work, but this seems close enough to move forward once the questions above are addressed.

…olidate gitignore

jameslamb · 2024-10-29T20:23:30Z

It seems like there are opportunities to move from CUDA 12.1 to 12.4 and drop the pytorch channel (in favor of conda-forge) in future work, but this seems close enough to move forward once the questions above are addressed.

Yep absolutely! That's a good next step. Thanks for reviewing, I do think we should move it forward once you feel like all your questions have been addressed. The sooner we get CI fully working in this repo, the sooner we can cut all this stuff out of the cugraph / wholegraph repos and work on such problems here, in one place.

bdice · 2024-10-29T20:36:35Z

ci/run_cugraph_pyg_pytests.sh

@@ -9,7 +9,7 @@ cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")"/../python/cugraph-pyg/cugraph_
 pytest --cache-clear --benchmark-disable "$@" .

 # Used to skip certain examples in CI due to memory limitations
-export CI_RUN=1
+export CI=true


If we export this, it means that local executions of this script will also skip those tests. I'm not sure if that is really intended or not.

I think the cugraph developers intended that. @alexbarghi-nv introduced that in rapidsai/cugraph#4384

jameslamb · 2024-10-30T13:04:49Z

/merge

Closes #53 This pulls over the remaining changes from that PR... thought that was easier than fixing the merge conflicts between that and recent PRs. * adds notebook testing * adds all the remaining `build.sh` changes from #53 (mostly for interactive development) * adds `cugraph-pyg` and `cugraph-dgl` tests in `conda-python-tests` (missed in #59) ## Notes for Reviewers I left other comments explaining these changes inline. Authors: - James Lamb (https://github.com/jameslamb) - Tingyu Wang (https://github.com/tingyu66) Approvers: - Tingyu Wang (https://github.com/tingyu66) - Ray Douglass (https://github.com/raydouglass) URL: #62

As the issue around PyTorch being built without NumPy was fixed in conda-forge, we can now relax these upper bounds to allow PyTorch 2.4. xref: conda-forge/pytorch-cpu-feedstock#254 xref: conda-forge/pytorch-cpu-feedstock#266 xref: rapidsai/cugraph#4615 xref: rapidsai/cugraph#4703 xref: #59 Authors: - https://github.com/jakirkham Approvers: - Jake Awe (https://github.com/AyodeAwe) - Tingyu Wang (https://github.com/tingyu66) URL: #75

add PR CI for cugraph-pyg and cugraph-dgl

74b5c95

jameslamb added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Oct 21, 2024

jameslamb added 3 commits October 21, 2024 11:08

remove some duplication in dependencies.yaml

8b80338

fix env for conda builds, stopping have builds depend on tests

fb5e9e0

fix wheel metadata

84fccbc

jameslamb added the DO NOT MERGE Hold off on merging; see PR for details label Oct 21, 2024

jameslamb added 7 commits October 21, 2024 12:44

cugraph-pyg and cugraph-dgl have runtime dependencies on pandas... th…

74ecbe5

…ey should express them

remove test_wheel.sh script

2ed9905

declare torch testing dependency for cugraph-dgl, combine pip install…

802e46c

…s again

use pytorch_geometric instead of pyg

4033210

the pip package is called 'torch-geometric' not 'pytorch_geometric'

fad3e8b

update dgl doc, include karata.csv

887e32f

update to pytorch and dgl pins used in cugraph repo

4344fd0

jameslamb commented Oct 21, 2024

View reviewed changes

ci/test_wheel_cugraph-dgl.sh Show resolved Hide resolved

alexbarghi-nv added 2 commits October 22, 2024 13:50

update files

b86bdd3

update view.py

7897f2f

alexbarghi-nv removed the DO NOT MERGE Hold off on merging; see PR for details label Oct 25, 2024

jameslamb mentioned this pull request Oct 28, 2024

Pin RAPIDS nightly version in conda installs rapidsai/build-planning#106

Closed

jameslamb added 3 commits October 28, 2024 10:33

Merge branch 'branch-24.12' into add-pyg-and-dgl-ci

2c0e654

set RAPIDS_DATASET_ROOT_DIR

e718840

re-enable all CI, remove unintentionally-included cufile.log

7442922

jameslamb changed the title ~~WIP: [DO NOT MERGE] add PR CI for cugraph-pyg and cugraph-dgl~~ add PR CI for cugraph-pyg and cugraph-dgl Oct 28, 2024

jameslamb requested a review from alexbarghi-nv October 28, 2024 21:32

jameslamb marked this pull request as ready for review October 28, 2024 21:32

jameslamb requested review from a team as code owners October 28, 2024 21:32

jameslamb requested a review from raydouglass October 28, 2024 21:32

print sccache stats in builds

6cc3dec

jameslamb mentioned this pull request Oct 29, 2024

Print sccache stats in all builds rapidsai/build-planning#111

Closed

Merge branch 'add-pyg-and-dgl-ci' of github.com:jameslamb/cugraph-gnn…

a421fc9

… into add-pyg-and-dgl-ci

alexbarghi-nv approved these changes Oct 29, 2024

View reviewed changes

bdice reviewed Oct 29, 2024

View reviewed changes

jameslamb added 2 commits October 29, 2024 13:07

remove unnecessary channel for builds, use CI instead of CI_RUN, cons…

1202c36

…olidate gitignore

more dataset to datasets

e6c07fa

bdice approved these changes Oct 29, 2024

View reviewed changes

rapids-bot bot merged commit 31ee98f into rapidsai:branch-24.12 Oct 30, 2024
78 checks passed

jameslamb deleted the add-pyg-and-dgl-ci branch October 30, 2024 13:05

jameslamb mentioned this pull request Nov 4, 2024

add notebook tests, build.sh args #62

Merged

jakirkham mentioned this pull request Nov 21, 2024

Relax PyTorch upper bound (allowing 2.4) #75

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add PR CI for cugraph-pyg and cugraph-dgl #59

add PR CI for cugraph-pyg and cugraph-dgl #59

jameslamb commented Oct 21, 2024 •

edited

Loading

jameslamb commented Oct 28, 2024

alexbarghi-nv left a comment

bdice Oct 29, 2024

jameslamb Oct 29, 2024

bdice Oct 29, 2024

jameslamb Oct 29, 2024

bdice Oct 29, 2024

jameslamb Oct 29, 2024

bdice commented Oct 29, 2024

jameslamb commented Oct 29, 2024

bdice Oct 29, 2024

jameslamb Oct 29, 2024

jameslamb commented Oct 30, 2024

		# Used to skip certain examples in CI due to memory limitations
		export CI_RUN=1

		dataset/
		datasets/

add PR CI for cugraph-pyg and cugraph-dgl #59

add PR CI for cugraph-pyg and cugraph-dgl #59

Conversation

jameslamb commented Oct 21, 2024 • edited Loading

Notes for Reviewers

jameslamb commented Oct 28, 2024

alexbarghi-nv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice commented Oct 29, 2024

jameslamb commented Oct 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jameslamb commented Oct 30, 2024

jameslamb commented Oct 21, 2024 •

edited

Loading