DGL Dataloader #3181

VibhuJawa · 2023-01-25T06:42:48Z

This PR depends upon ~~#3148~~ (#3170)

This PR closes #3154

Works on SG Homogenous Graph (OBGN-Products)
Works on SG Heterogenous Graphs (OBGN-Mag)
Works on MG Homogenous Graphs (OBGN-Products, OBGN-PAPERS100M (1 Billion edge graph))
Works on MG Heterogenous Graphs (OBGN-Mag)

Tests to add:

Tests on SG Homogenous Graphs
Tests on SG Heterogenous Graphs
Tests on MG Homogenous Graphs
Tests on MG Heterogenous Graphs

Maybe Todo:
~~- [ ] Test Multi-trainer examples and verification~~ (Will do a follow up)

…AT scale option.

…2-initial_benchmark_reorg

…to cp-offsets

This reverts commit 2f99d6e.

rlratzel

Thanks for all this hard work. I noticed some minor things.

I also wish we didn't have to copy DataFrames for the reverse_edges code, but I can't think of a more efficient way either.

python/cugraph-dgl/cugraph_dgl/cugraph_storage.py

python/cugraph-dgl/cugraph_dgl/dataloading/dataloader.py

rlratzel · 2023-01-27T23:28:08Z

python/cugraph-dgl/cugraph_dgl/dataloading/dataset.py

+        if self._input_directory is None:
+            raise dgl.DGLError(
+                "Please set input directory by calling `set_input_directory` "
+                "before trying to fetch a sample"
+            )


Rather than this check and another method, can the ctor just require an input_directory arg and do the set_input_directory steps itself upfront?

So i did not do this because this is set/reset per epoch basis as each epoch has a different directory, Happy to discuss if you have ideas on how to do this cleanly.

Let me know if this is fine for now please.

python/cugraph-dgl/cugraph_dgl/dataloading/neighbor_sampler.py

rlratzel · 2023-01-30T05:15:31Z

python/cugraph/cugraph/experimental/__init__.py

+
+from cugraph.gnn.data_loading import EXPERIMENTAL__BulkSampler
+
+BulkSampler = experimental_warning_wrapper(EXPERIMENTAL__BulkSampler)


FYI PR #3170 changed this so that the gnn namespace wouldn't be lost, based on thisfeedback.

Yup, waiting on #3170 to land before i make these changes.

VibhuJawa · 2023-01-30T19:57:55Z

I also wish we didn't have to copy DataFrames for the reverse_edges code, but I can't think of a more efficient way either.

A couple of clarifications:

We need to copy frames to add the edge_ids column and not for reverse_edges
For non persisted dask-cudf frames this does not return in any memory overhead .
For persisted cudf Frames/dask-cudf Frames we will in future enable COW(copy-on-write) [REVIEW] Merge copy-on-write feature branch into branch-23.04 cudf#12619 which will i think will mean that we have non memory overhead for this sort of behavior going forward.

CC: @galipremsagar , For context we are renaming some columns and adding an extra column but can not modify the input DataFrame. I think your amazing work with COW will address these cases.

galipremsagar · 2023-01-30T20:25:03Z

For persisted cudf Frames/dask-cudf Frames we will in future enable COW(copy-on-write) [REVIEW] Merge copy-on-write feature branch into branch-23.02 cudf#12619 which will i think will mean that we have non memory overhead for this sort of behavior going forward.

CC: @galipremsagar , For context we are renaming some columns and adding an extra column but can not modify the input DataFrame. I think your amazing work with COW will address these cases.

Yes, COW will remove the memory overhead. But we are slipping it to 23.04. For now you could use shallow copies, if that doesn't suffice, happy to hop on a call to discuss some internal API solutions offline.

…uJawa/cugraph into bulk_sampler_dataloader_dgl

VibhuJawa · 2023-01-31T06:08:29Z

python/cugraph/cugraph/dask/sampling/uniform_neighbor_sample.py

-    ).persist()
+    empty_df = (
+        create_empty_df_with_edge_props(indices_t, weight_t)
+        if with_edge_properties
+        else create_empty_df(indices_t, weight_t)
+    )
+    ddf = dask_cudf.from_delayed(result, meta=empty_df, verify_meta=False).persist()


Will be over-writted by : #3170 so please ignore.

VibhuJawa · 2023-01-31T06:08:58Z

python/cugraph/cugraph/gnn/data_loading/__init__.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from cugraph.gnn.data_loading.bulk_sampler import EXPERIMENTAL__BulkSampler


Same for this file #3170

VibhuJawa · 2023-01-31T06:32:30Z

CC: @rlratzel , Have resolved all changes that were part of this PR , other minor 3 variable name changes will come once 3180 lands.

codecov-commenter · 2023-01-31T08:48:53Z

Codecov Report

Base: 55.30% // Head: 55.61% // Increases project coverage by +0.31% 🎉

Coverage data is based on head (e37aa69) compared to base (068629e).
Patch coverage: 74.12% of modified lines in pull request are covered.

Additional details and impacted files

@@               Coverage Diff                @@
##           branch-23.02    #3181      +/-   ##
================================================
+ Coverage         55.30%   55.61%   +0.31%     
================================================
  Files               148      150       +2     
  Lines              9573     9671      +98     
================================================
+ Hits               5294     5379      +85     
- Misses             4279     4292      +13

Impacted Files	Coverage Δ
...ugraph/cugraph/dask/structure/mg_property_graph.py	`11.90% <0.00%> (-0.02%)`	⬇️
...ure/graph_implementation/simpleDistributedGraph.py	`14.24% <0.00%> (-0.09%)`	⬇️
...h/cugraph/dask/sampling/uniform_neighbor_sample.py	`24.75% <41.17%> (+4.10%)`	⬆️
.../cugraph/dask/centrality/eigenvector_centrality.py	`30.00% <50.00%> (ø)`
...cugraph/cugraph/dask/centrality/katz_centrality.py	`19.56% <50.00%> (ø)`
python/cugraph/cugraph/dask/community/egonet.py	`18.51% <50.00%> (ø)`
python/cugraph/cugraph/dask/community/louvain.py	`30.30% <50.00%> (ø)`
...n/cugraph/cugraph/dask/community/triangle_count.py	`21.62% <50.00%> (ø)`
...on/cugraph/cugraph/dask/components/connectivity.py	`31.03% <50.00%> (ø)`
python/cugraph/cugraph/dask/cores/core_number.py	`26.66% <50.00%> (ø)`
... and 14 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

rlratzel

Thanks for the updates. The use_ddp additions are pretty exciting.

rlratzel · 2023-01-31T15:51:37Z

/merge

rlratzel and others added 30 commits December 14, 2022 06:17

Using MultiGraph instead of Graph for benchmarks, added additional RM…

f650429

…AT scale option.

Merge remote-tracking branch 'upstream/branch-23.02' into branch-23.0…

5d0bd77

…2-initial_benchmark_reorg

fixes

b85f9fd

fix merge conflict

3bcdbb3

persist

c9b30c4

remove unwanted files

e1e410d

update result object creation

296fdfa

fix style

4c87528

fix broken mg sampling test

67b464b

fix sg pyg tests

447d9ee

config pyg, dgl work

96e2ac3

update

af57f08

print array size for debugging

f563945

fix merge conflict

19376b5

Added cugraph+UCX build instructions

d513a86

pg dtype bugfix

cb791ac

fix mg dtype bug:

1af9495

fix formatting issue

5670bed

add ci test skip

6d6dfea

fix renumbering test

16fc189

fix tests

6bd78e6

Merge branch 'branch-23.02' of https://github.com/rapidsai/cugraph in…

99029b0

…to cp-offsets

ADD DGL benchmarks

2f99d6e

Revert "ADD DGL benchmarks"

9c41ece

This reverts commit 2f99d6e.

remove unwanted files

91c1a3a

remove debug code

8475792

drop persist from get_vertex_data, change dtype only if necessary

2679be0

propagate change to get_edge_data

6b1dbd8

add type check, explanatory comments

8734e31

docstring fix

88dac25

VibhuJawa added 7 commits January 28, 2023 01:57

Merge branch 'branch-23.02' into make_graph_objects_serializable

c085c94

Merge branch 'branch-23.02' into bulk_sampler_dataloader_dgl

d5b090d

Merge branch 'branch-23.02' into make_graph_objects_serializable

f300e3b

Remove orphaned comment

0cab2ef

Merge branch 'branch-23.02' into make_graph_objects_serializable

457be7b

Update uniform_neighbor_sample.py

cfa8625

Minor cleanup after merge

49bfd71

rlratzel requested changes Jan 30, 2023

View reviewed changes

BradReesWork added the merge conflict label Jan 30, 2023

Merge branch 'branch-23.02' into make_graph_objects_serializable

8bab0e4

VibhuJawa added 8 commits January 31, 2023 01:59

Merge branch 'branch-23.02' into bulk_sampler_dataloader_dgl

195062a

Update uniform_neighbor_sample.py

d8e9092

POC for multi-trainers cugraph-DGL

fb28f88

Merge branch 'branch-23.02' into bulk_sampler_dataloader_dgl

9694a52

POC for multi-trainers cugraph-DGL

d22b258

Address Reviews

4a29a09

Address Reviews

d9e9c56

Merge branch 'bulk_sampler_dataloader_dgl' of https://github.com/Vibh…

f34cfd3

…uJawa/cugraph into bulk_sampler_dataloader_dgl

VibhuJawa commented Jan 31, 2023

View reviewed changes

VibhuJawa added 2 commits January 30, 2023 22:14

Address Reviews

68082ec

bug fix with MG sampling

e37aa69

VibhuJawa removed the merge conflict label Jan 31, 2023

rlratzel approved these changes Jan 31, 2023

View reviewed changes

rapids-bot bot merged commit a51bfa7 into rapidsai:branch-23.02 Jan 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DGL Dataloader #3181

DGL Dataloader #3181

VibhuJawa commented Jan 25, 2023 •

edited

Loading

rlratzel left a comment

rlratzel Jan 27, 2023

VibhuJawa Jan 30, 2023

VibhuJawa Jan 31, 2023

rlratzel Jan 30, 2023

VibhuJawa Jan 30, 2023

VibhuJawa commented Jan 30, 2023

galipremsagar commented Jan 30, 2023

VibhuJawa Jan 31, 2023

VibhuJawa Jan 31, 2023

VibhuJawa commented Jan 31, 2023

codecov-commenter commented Jan 31, 2023 •

edited

Loading

rlratzel left a comment

rlratzel commented Jan 31, 2023


		from cugraph.gnn.data_loading import EXPERIMENTAL__BulkSampler

		BulkSampler = experimental_warning_wrapper(EXPERIMENTAL__BulkSampler)

DGL Dataloader #3181

DGL Dataloader #3181

Conversation

VibhuJawa commented Jan 25, 2023 • edited Loading

rlratzel left a comment

Choose a reason for hiding this comment

rlratzel Jan 27, 2023

Choose a reason for hiding this comment

VibhuJawa Jan 30, 2023

Choose a reason for hiding this comment

VibhuJawa Jan 31, 2023

Choose a reason for hiding this comment

rlratzel Jan 30, 2023

Choose a reason for hiding this comment

VibhuJawa Jan 30, 2023

Choose a reason for hiding this comment

VibhuJawa commented Jan 30, 2023

galipremsagar commented Jan 30, 2023

VibhuJawa Jan 31, 2023

Choose a reason for hiding this comment

VibhuJawa Jan 31, 2023

Choose a reason for hiding this comment

VibhuJawa commented Jan 31, 2023

codecov-commenter commented Jan 31, 2023 • edited Loading

Codecov Report

rlratzel left a comment

Choose a reason for hiding this comment

rlratzel commented Jan 31, 2023

VibhuJawa commented Jan 25, 2023 •

edited

Loading

codecov-commenter commented Jan 31, 2023 •

edited

Loading