Optimize K-Truss #4375

jnke2016 · 2024-04-25T16:11:39Z

This PR leverages our existing edge triangle count to implement both SG and MG K-Truss as our initial version.
This PR also:

Exposes the rx count in several as our shuffling functions
Add C and C++ tests for MG K-Truss

Closes #4500

…6_optimize-k-truss

…o_local_gpu_by_vertex_partitioning'

ChuckHastings

So much simpler. A few cleanup comments, otherwise I think it's looking pretty good.

ChuckHastings · 2024-07-24T17:41:23Z

cpp/src/community/edge_triangle_count_impl.cuh

@@ -136,7 +136,7 @@ edge_property_t<graph_view_t<vertex_t, edge_t, false, multi_gpu>, edge_t> edge_t
  auto edge_first = thrust::make_zip_iterator(edgelist_srcs.begin(), edgelist_dsts.begin());

  size_t edges_to_intersect_per_iteration =
-    static_cast<size_t>(handle.get_device_properties().multiProcessorCount) * (1 << 17);
+    static_cast<size_t>(handle.get_device_properties().multiProcessorCount) * (1 << 13);


Is there a reason for 13? Should this be some sort of named constant so it's easier to change?

It is a good idea to expose it in order to easily control the chunk size. I temporarily set a lower value so that I can process a larger graph when running at scale

cpp/include/cugraph/detail/shuffle_wrappers.hpp

ChuckHastings · 2024-07-24T17:42:20Z

cpp/src/community/k_truss_mg_v32_e32.cu

@@ -0,0 +1,41 @@
+/*
+ * Copyright (c) 2023-2024, NVIDIA CORPORATION.


ChuckHastings · 2024-07-24T17:42:33Z

cpp/src/community/k_truss_mg_v32_e64.cu

@@ -0,0 +1,41 @@
+/*
+ * Copyright (c) 2023-2024, NVIDIA CORPORATION.


cpp/src/community/k_truss_mg_v64_e64.cu

ChuckHastings · 2024-07-24T17:48:41Z

cpp/src/community/k_truss_impl.cuh


-    std::tie(edgelist_srcs, edgelist_dsts, edgelist_wgts, num_triangles, std::ignore) =
-      decompress_to_edgelist(
+    std::chrono::seconds s (0);             // 1 second


Two thoughts...

Have you looked at https://github.com/rapidsai/cugraph/blob/branch-24.08/cpp/include/cugraph/utilities/high_res_timer.hpp which I think you could just include and use rather than reinventing. There are features to manage multiple timers inside that as well

Probably should remove these before we merge the PR

I only wanted to time the triangle count part and I got a compiler error when copying the high_res_timer timing code from our tests so I didn't spend much time understanding what was wrong. But I will look into it again.

Probably should remove these before we merge the PR

I only have them for benchmarking on Draco. I will remove them in my next commit today along with all the unused functors

cpp/tests/CMakeLists.txt

seunghwak · 2024-07-25T19:20:43Z

cpp/tests/community/mg_k_truss_test.cpp

+struct KTruss_Usecase {
+  int32_t k_{3};
+  bool test_weighted_{false};
+  // FIXME: test edge mask


Any reason we are skipping this? (edge mask)

Oh I am not. I just forgot to remove this outdated comment

seunghwak · 2024-07-25T19:24:08Z

cpp/tests/community/mg_k_truss_test.cpp

+        mg_graph_view.vertex_partition_range_lasts());
+
+      auto global_d_cugraph_srcs = cugraph::test::device_gatherv(
+        *handle_, raft::device_span<vertex_t const>(d_cugraph_srcs.data(), d_cugraph_srcs.size()));


Sorry for additional nitpicking, but it might be better to follow naming convention of K-core tests (https://github.com/rapidsai/cugraph/blob/branch-24.08/cpp/tests/cores/mg_k_core_test.cpp#L136). Later, if this pattern occurs frequent enough, we can create another utility funciton.

seunghwak · 2024-07-25T19:29:02Z

cpp/src/community/k_truss_impl.cuh

@@ -497,7 +132,7 @@ k_truss(raft::handle_t const& handle,
                                            exclude_self_loop_t<vertex_t>{});

    if constexpr (multi_gpu) {
-      std::tie(srcs, dsts, std::ignore, std::ignore, std::ignore) =
+      std::tie(srcs, dsts, std::ignore, std::ignore, std::ignore, std::ignore) =


You can defer this to the next MG optimization PR, but you can use masking here instead of creating a new graph for excluding self-loops (see the example here. https://github.com/rapidsai/cugraph/blob/branch-24.08/cpp/src/community/triangle_count_impl.cuh#L351).

seunghwak · 2024-07-25T19:30:53Z

cpp/src/community/k_truss_impl.cuh

@@ -656,265 +291,53 @@ k_truss(raft::handle_t const& handle,
                                                   *vertex_partition_range_lasts);
    }
    renumber_map = std::move(tmp_renumber_map);
+


You can use masking instead of creating a graph except for the final DODG graph creation (this DODG graph will be used again & again so better create a new graph than using masking as masking has its own overhead).

seunghwak · 2024-07-25T19:34:25Z

cpp/src/community/k_truss_impl.cuh

+    while (true) {
+
+      auto edge_triangle_counts =
+        edge_triangle_count<vertex_t, edge_t, multi_gpu>(handle, cur_graph_view);


Add a FIXME statement here.

This approach will miserably fail if we need to go through many iterations and only a small number of edges are invalidated in each iteration. No need to address this if this doesn't happen for any practical graphs but if we find one, we can work on optimization focusing on that graph.

…6_optimize-k-truss

seunghwak

LGTM

…6_optimize-k-truss

ChuckHastings · 2024-07-30T06:01:38Z

/merge

enable k-1 core

be543b1

github-actions bot added the cuGraph label Apr 25, 2024

jnke2016 added 4 commits April 26, 2024 09:30

perform edge triangle count in chunk

327a07e

add edge triangle count tests

1f00dd6

move edge triangle count to the stable API

da330b3

fix style

913283d

github-actions bot added the CMake label May 4, 2024

jnke2016 added 23 commits May 5, 2024 01:16

update function definition

9d2d4f7

fix style

2019d99

update test

701e33d

return edge_property_t

0d74246

fix style

c103e52

add edge mask tests

50798d5

fix style

86fd201

add mg implementation of edge triangle count

aad7590

add reference for mg edge triangle count

28149f7

add mg edge triangle count tests

0e69382

remove debug print and unused import

f893d24

add edge mask test

30f891a

update 'mg_graph_to_sg_graph' to support 'edge_ids'

be7ed1a

add fixme

e705eca

add doxygen documentation

2453a6f

explicitly provide template parameter types

e69f862

rename variable

2bb9cba

remove unnecessary sort

6a02f03

round with the raft util function

17017ec

update fixme

13501fe

rename variable

e5a0f2d

fix style

ca30c84

Merge remote-tracking branch 'upstream/branch-24.06' into branch-24.0…

ba48f90

…6_optimize-k-truss

jnke2016 added 5 commits July 18, 2024 14:11

update branch

a0eb24f

enable edge masking for k-core and k-truss and add tests

a34cef3

enable int64 type for 'd_values' in 'shuffle_int_vertex_value_pairs_t…

b1aeab4

…o_local_gpu_by_vertex_partitioning'

fix type bug

533f374

update benchmark tests and simplify initial k-truss implementation

ae17245

ChuckHastings reviewed Jul 24, 2024

View reviewed changes

jnke2016 added 6 commits July 25, 2024 07:50

reset chunk parameter

2733cec

remove debug statement

6de56d5

update CMake file

5892e44

describe rx_count in documentation

c639b76

update copyright

48078e9

fix copyright

f88efb9

seunghwak reviewed Jul 25, 2024

View reviewed changes

jnke2016 added 7 commits July 25, 2024 12:55

remove outdated fixme

f743122

update docs

0c2f042

add fixme

94a4c89

enable MG CAPI k_truss

4476d8d

add CAPI tests for MG k_truss

ed78d32

fix style

6de5304

Merge remote-tracking branch 'upstream/branch-24.08' into branch-24.0…

61c6797

…6_optimize-k-truss

seunghwak approved these changes Jul 29, 2024

View reviewed changes

ChuckHastings approved these changes Jul 29, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/branch-24.08' into branch-24.0…

3743208

…6_optimize-k-truss

jnke2016 force-pushed the branch-24.06_optimize-k-truss branch from d96081c to 3743208 Compare July 29, 2024 21:07

rapids-bot bot merged commit a1f8a65 into rapidsai:branch-24.08 Jul 30, 2024
132 checks passed

bdice removed request for a team July 30, 2024 06:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize K-Truss #4375

Optimize K-Truss #4375

jnke2016 commented Apr 25, 2024 •

edited

Loading

ChuckHastings left a comment

ChuckHastings Jul 24, 2024

jnke2016 Jul 24, 2024

ChuckHastings Jul 24, 2024

ChuckHastings Jul 24, 2024

ChuckHastings Jul 24, 2024

jnke2016 Jul 24, 2024

seunghwak Jul 25, 2024

jnke2016 Jul 25, 2024

seunghwak Jul 25, 2024

seunghwak Jul 25, 2024

seunghwak Jul 25, 2024

seunghwak Jul 25, 2024

seunghwak left a comment

ChuckHastings commented Jul 30, 2024

		@@ -0,0 +1,41 @@
		/*
		* Copyright (c) 2023-2024, NVIDIA CORPORATION.

Optimize K-Truss #4375

Optimize K-Truss #4375

Conversation

jnke2016 commented Apr 25, 2024 • edited Loading

ChuckHastings left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seunghwak left a comment

Choose a reason for hiding this comment

ChuckHastings commented Jul 30, 2024

jnke2016 commented Apr 25, 2024 •

edited

Loading