Update Louvain to use new graph primitives and pattern accelerators #1423

ChuckHastings · 2021-02-23T01:15:33Z

Implement the update_by_delta_modularity method using the new graph primitives and pattern accelerators.

This eliminates all of the custom MNMG implementation originally created for MNMG Louvain a few releases ago and replaces it with the new pattern accelerator and graph primitives that have been added in the last couple of releases.

This depends on the following PRs and should not be merged until after them:

closes #1220

1) Fix Dendrogram methods to be const consistent 2) Update experimental update_by_delta_modularity to use new graph primitives. Clean up unused local code

cpp/src/experimental/louvain.cuh

seunghwak · 2021-03-16T14:34:08Z

Did you merge with the latest cuGraph branch? I am seeing some updates I made in "Files Changed"

ChuckHastings · 2021-03-16T15:02:04Z

Did you merge with the latest cuGraph branch? I am seeing some updates I made in "Files Changed"

Just double checked. I did merge them.

I had to make a couple of additional changes from my debugging. There was a problem in device_comm.cuh, I suspect I'm the first code that actually compiles the device_sendrecv_tuple_iterator_element_impl code in this way.

I modified generate_graph_from_edgelist to allow us to skip renumbering. 1 level of Louvain is deterministic if the graphs are numbered the same. This allows me to compare single GPU and multi GPU results to validate behavior of Louvain. If renumbering is on there's no good way to compare results across runs. The renumbering result is a function of the number of GPUs.

seunghwak · 2021-03-16T14:38:16Z

cpp/src/community/dendrogram.cuh

@@ -26,12 +26,14 @@ template <typename vertex_t>
 class Dendrogram {
 public:
  void add_level(vertex_t num_verts,
+                 vertex_t first_index,


I think add_level(vertex_t first_index, vertex_t num_(local_)verts, ...) is more idiomatic than add_level(vertex_t num_verts, vertex_t first_index, ...)

seunghwak · 2021-03-16T14:39:06Z

cpp/src/community/dendrogram.cuh

@@ -26,12 +26,14 @@ template <typename vertex_t>
 class Dendrogram {
 public:
  void add_level(vertex_t num_verts,
+                 vertex_t first_index,
                 cudaStream_t stream                 = 0,
                 rmm::mr::device_memory_resource *mr = rmm::mr::get_current_device_resource())
  {
    level_ptr_.push_back(
      std::make_unique<rmm::device_buffer>(num_verts * sizeof(vertex_t), stream, mr));


Any reason for using rmm::device_buffer instead of rmm::device_uvector?

The code I copied this from used device_buffer...

Can we type erase and pass a device_uvector back to python? This is currently C++ only, but I was intending that eventually the python interface would be able to fetch these values and store them in cudf columns.

@Iroy30 may be able to confirm, but I guess this is possible.

https://github.com/rapidsai/cugraph/blob/branch-0.19/cpp/include/utilities/cython.hpp#L21

rmm::device_uvector is widely used in cython.hpp.

Looks like I can convert a device_uvector into a device_buffer when it's needed in python:

cugraph/cpp/include/utilities/cython.hpp

Line 200 in 7256f32

return std::make_pair(std::make_unique<rmm::device_buffer>(dv_.release()), sizeof(vertex_t));

I'll pursue this change.

seunghwak · 2021-03-16T15:19:26Z

cpp/src/experimental/louvain.cuh

 #include <experimental/graph_functions.hpp>
+
 #include <patterns/copy_to_adj_matrix_row_col.cuh>
 #include <patterns/copy_v_transform_reduce_in_out_nbr.cuh>


Do you still need this?

The blank line? Perhaps not. It prevents clang-format from reordering includes across blocks, I felt like the patterns should be after the graph and functions definitions... but I haven't verified that it's required.

No, #include <patterns/copy_v_transform_reduce_in_out_nbr.cuh>

AFAIK, this was used to compute weight_sums which has been replaced by a graph_view_t class member function.

Ah, yes. You are correct. I have deleted that.

Spoke too soon. I actually still need it for a different purpose.

cugraph/cpp/src/experimental/louvain.cuh

Line 316 in 9e4421e

experimental::copy_v_transform_reduce_out_nbr(

seunghwak · 2021-03-16T15:21:28Z

cpp/src/experimental/louvain.cuh

+      vertex_weights_v_(graph_view.get_number_of_local_vertices(), handle.get_stream()),
+      src_vertex_weights_cache_v_(0, handle.get_stream()),
+      src_cluster_cache_v_(0, handle.get_stream()),
+      dst_cluster_cache_v_(0, handle.get_stream()),
      stream_(handle.get_stream())


What's the purpose of having both handle_ and stream_ as stream is just handle_.get_stream()? This is like maintaining two copies of the same variable.

The code was adapted from a previous version before we had handle. I can replace the stream_ references with handle_.get_stream()

cpp/src/experimental/louvain.cuh

seunghwak · 2021-03-16T15:30:07Z

cpp/src/experimental/louvain.cuh

+    auto dst = cache_dst_vertex_properties(input, dst_cache_v);
+
+    return std::make_tuple(src, dst);
+  }


Do we really need this function anymore? This may cut one line in the caller site but just add an additional layer of indirection to track.

Deleted this. In my original MNMG implementation this was called dozens of times.

seunghwak · 2021-03-16T15:31:57Z

cpp/src/experimental/louvain.cuh

+                                         resolution * (a_new * k_k - a_old * k_k + k_k * k_k) /
+                                           (total_edge_weight * total_edge_weight));
+
+#if 0


Better delete temporary print code.

Thanks. Hadn't promoted this from draft yet, that was on my todo list.

seunghwak · 2021-03-16T15:32:16Z

cpp/src/experimental/louvain.cuh

+        vertex_t new_cluster      = thrust::get<0>(p);
+        weight_t delta_modularity = thrust::get<1>(p);
+
+#if 0


Better delete temporary print code.

seunghwak · 2021-03-16T15:34:01Z

cpp/tests/community/louvain_mg_test.cpp

@@ -0,0 +1,189 @@
+/*


I recommend renaming this to mg_louvain_test.cpp as the executable name is MG_LOUVAIN_TEST.

seunghwak · 2021-03-16T15:35:42Z

cpp/tests/community/louvain_mg_test.cpp

+// will be instantiated as the parameter to the tests defined below using
+// INSTANTIATE_TEST_CASE_P()
+//
+struct Louvain_Testparams {


We're mostly using AnalyticsName_Usecase_t, what's the purpose of introducing a new naming convention?

Changed Testparams to Usecase

seunghwak · 2021-03-16T15:43:46Z

I modified generate_graph_from_edgelist to allow us to skip renumbering. 1 level of Louvain is deterministic if the graphs are numbered the same. This allows me to compare single GPU and multi GPU results to validate behavior of Louvain. If renumbering is on there's no good way to compare results across runs. The renumbering result is a function of the number of GPUs.

You mean the outcome is dependent on vertex ordering?

ChuckHastings · 2021-03-16T18:12:45Z

I modified generate_graph_from_edgelist to allow us to skip renumbering. 1 level of Louvain is deterministic if the graphs are numbered the same. This allows me to compare single GPU and multi GPU results to validate behavior of Louvain. If renumbering is on there's no good way to compare results across runs. The renumbering result is a function of the number of GPUs.

You mean the outcome is dependent on vertex ordering?

Yes. The algorithm breaks ties by choosing the vertex with the smaller vertex id making it deterministic with consistent vertex ordering.

seunghwak · 2021-03-16T18:51:39Z

Yes. The algorithm breaks ties by choosing the vertex with the smaller vertex id making it deterministic with consistent vertex ordering.

This sounds against graph invariants (https://en.wikipedia.org/wiki/Graph_property) we may try to preserve. Isn't there a better way to break a tie without violating graph invariants? (yeah... this sounds impossible; if there are two structurally equivalent communities to move in... no way to distinguish the two without relying on labeling...)

What's the purpose of providing determinism? Just for testing or there is a bigger reason?

ChuckHastings · 2021-03-16T19:02:37Z

Yes. The algorithm breaks ties by choosing the vertex with the smaller vertex id making it deterministic with consistent vertex ordering.

This sounds against graph invariants (https://en.wikipedia.org/wiki/Graph_property) we may try to preserve. Isn't there a better way to break a tie without violating graph invariants? (yeah... this sounds impossible; if there are two structurally equivalent communities to move in... no way to distinguish the two without relying on labeling...)

What's the purpose of providing determinism? Just for testing or there is a bigger reason?

Ease of testing. Request from cuML for it to be deterministic.

Note that Louvain approximates optimizing modularity, exact solution is NP hard. The serial implementation of Louvain randomly orders the vertices before each pass. Consequently it randomly picks one of clusters when it encounters a tie. The consequence of this is that every time you run Louvain you get a different approximation of the answer.

seunghwak · 2021-03-16T19:35:31Z

Ease of testing. Request from cuML for it to be deterministic.

Let me think more but I am little worried about the current testing approach as this bypasses all the optimizations relying on renumbering; and leaves those optimizations untested.

The current best approach I can think of is to 1) create a renumbered multi_gpu graph, 2) and using the renumber_map from here, renumber edge list to be used in creating a single_gpu graph, 3) and create a single_gpu graph with renumbering disabled.

The results will be deterministic and SG & MG results will coincide, and we can compare optimization disabled path of SG implementation with optimization enabled path of MG implementation, so this will have a better coverage.

ChuckHastings · 2021-03-16T19:40:46Z

Ease of testing. Request from cuML for it to be deterministic.

Let me think more but I am little worried about the current testing approach as this bypasses all the optimizations relying on renumbering; and leaves those optimizations untested.

The current best approach I can think of is to 1) create a renumbered multi_gpu graph, 2) and using the renumber_map from here, renumber edge list to be used in creating a single_gpu graph, 3) and create a single_gpu graph with renumbering disabled.

The results will be deterministic and SG & MG results will coincide, and we can test optimization disabled path of SG implementation with optimization enabled path of MG implementation, so this will have a better coverage.

Each renumbering pass in the SG and MG code would diverge. The results would be deterministic within a dendrogram level. But as soon as we coarsen the graph, the SG coarsen and the MG coarsen (with num GPUs > 1) will create different graphs and the results will diverge.

I had thought a bit about the fact that we're not testing the renumbering phase. We can certainly create some tests to specifically test the graph coarsening logic so that we're sure that's tested independently of the Louvain testing. But I can't come up with a good way to test that.

seunghwak · 2021-03-16T20:11:15Z

Ease of testing. Request from cuML for it to be deterministic.

Let me think more but I am little worried about the current testing approach as this bypasses all the optimizations relying on renumbering; and leaves those optimizations untested.
The current best approach I can think of is to 1) create a renumbered multi_gpu graph, 2) and using the renumber_map from here, renumber edge list to be used in creating a single_gpu graph, 3) and create a single_gpu graph with renumbering disabled.
The results will be deterministic and SG & MG results will coincide, and we can test optimization disabled path of SG implementation with optimization enabled path of MG implementation, so this will have a better coverage.

Each renumbering pass in the SG and MG code would diverge. The results would be deterministic within a dendrogram level. But as soon as we coarsen the graph, the SG coarsen and the MG coarsen (with num GPUs > 1) will create different graphs and the results will diverge.

I had thought a bit about the fact that we're not testing the renumbering phase. We can certainly create some tests to specifically test the graph coarsening logic so that we're sure that's tested independently of the Louvain testing. But I can't come up with a good way to test that.

So, there are two issues.

Testing with max_levels = 1
Testing when there are more than one levels (so graph coarsening will be called).

In case of 2), neither the proposed approach or the current approach will allow directly comparing SG & MG results.
And we're currently testing only the case 1). In the case of 1), both

1-1) completely skipping renumbering
and
1-2) renumbering edges using the renumber maps from the MG path before creating an SG graph

will work and allow directly comparing SG & MG results.

Are these correct? Assuming this, I think 1-2) has a better coverage than 1-1).

1. Exposed flatten_dendrogram and a version of louvain that returns dendrogram in algorithms 2. Added methods to the legacy graph implementations to make it easier to template between old and new graph objects 3. Removed some class member data that can be derived 4. Added code to suppress gtest output from rank > 0

codecov-io · 2021-03-25T17:12:35Z

Codecov Report

Merging #1423 (a8c91c2) into branch-0.19 (1f0f14e) will increase coverage by 0.02%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           branch-0.19    #1423      +/-   ##
===============================================
+ Coverage        58.24%   58.26%   +0.02%     
===============================================
  Files               71       71              
  Lines             3281     3283       +2     
===============================================
+ Hits              1911     1913       +2     
  Misses            1370     1370

Impacted Files	Coverage Δ
python/cugraph/_version.py	`44.80% <0.00%> (+0.39%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1f0f14e...a8c91c2. Read the comment docs.

afender

Lots of great stuff, I posted some nonblocking comments on API and testing.

afender · 2021-03-25T15:51:01Z

cpp/include/algorithms.hpp

+ *
+ * @tparam     graph_t               Type of graph
+ *
+ * @param[in]  handle                Library handle (RAFT). If a communicator is set in the handle,


"If a communicator is set in the handle, ..." missing the end of the sentence?

Removed extraneous text from copy/paste.

afender · 2021-03-25T15:52:21Z

cpp/include/algorithms.hpp

+ * @tparam     graph_t               Type of graph
+ *
+ * @param[in]  handle                Library handle (RAFT). If a communicator is set in the handle,
+ * @param[in]  graph                 input graph object (CSR)


We should start clarifying supported graphs (directed, distributed, self loops, multi edges)

cpp/include/algorithms.hpp

afender · 2021-03-25T16:01:04Z

cpp/src/community/louvain.cu

-                                    weight_t resolution)
+std::pair<std::unique_ptr<Dendrogram<vertex_t>>, weight_t> louvain(
+  raft::handle_t const &handle,
+  GraphCSRView<vertex_t, edge_t, weight_t> const &graph_view,


Isn't this the legacy one?

Yes. The legacy version is still faster on SG than the new version.

So, are we using the legacy graph for this? or are we using the legacy version for SG and the new version for MG?

The C++ supports:

Legacy graph for SG

new graph for SG

new graph for MG

The python is currently still using legacy graph for SG and new graph for MG. I'm doing some performance analysis now of the new graph for SG vice the legacy graph for SG. It might be necessary to get some optimizations in place in order for the new graph to be fast enough. My last run today (haven't pushed the latest mods yet, but will shortly) had web-Google.mtx running in 14s on the legacy implementation and 56s on the new implementation. ljournal-2008.mtx runs in 250s on legacy and crashes (OOM) on the new implementation.

Oops, ljournal is not large at all and if you see OOM with ljournal, I guess something really shouldn't happen is happening in this case; something we should take a look (maybe not in this PR).

I may dig deeper to both performance & memory footprint when I tune primitives for Louvain. The execution time and memory consumption sound very bad.

We should be able to process 1B-ish edge graph using one GPU. IIRC, ljournal is much smaller than that.

Agreed, that should probably be outside of this PR. But that's why I've left the legacy implementation in place and the default for python. When you start tuning these we can evaluate when to delete the legacy implementation.

afender · 2021-03-25T16:12:51Z

cpp/tests/community/mg_louvain_test.cpp

+INSTANTIATE_TEST_CASE_P(
+  simple_test,
+  Louvain_MG_Testfixture,
+  ::testing::Values(Louvain_Usecase("test/datasets/karate.mtx", true, 100, 1)


karate won't capture all cases. Other MG algos started to use RMAT #1438
That raises the question of comparing the output but we could at least make sure it runs successfully and returns a valid result by looking at the modularity score and range of values in the dendrogram on a larger graph.

Comparing the output should work for more than Karate. I didn't have enough time to verify other test cases and adding RMAT. I was hoping to expand later.

Sounds good, yeah this can go in the next PR along with python stuff etc.

afender · 2021-03-25T16:17:45Z

cpp/include/algorithms.hpp

@@ -14,10 +14,14 @@
 * limitations under the License.
 */
 #pragma once
+
+#include <dendrogram.hpp>


(question) for the python binding, are you planning on having a Louvain feature (maybe in cython.cu) that encapsulates the dendrogram aspects? Or will we add bindings for this class?

I was hoping to add bindings for a louvain that returns a Dendrogram, eventually. I had to make the C++ change to support MNMG testing which needed the Dendrogram exposed.

I like returning a dendrogram too (probably optionally).

Did you take a look at SLHC btw? It should return a dendrogram (might be in RAFT already). Better be consistent as we consider adding SLHC to cuGraph also.

Corey and I talked about the dendrogram and potentially moving it to raft. We thought waiting until the Louvain work was done in cugraph would be prudent.

seunghwak · 2021-03-25T22:08:06Z

cpp/src/community/louvain.cu

-                                    weight_t resolution)
+std::pair<std::unique_ptr<Dendrogram<vertex_t>>, weight_t> louvain(
+  raft::handle_t const &handle,
+  GraphCSRView<vertex_t, edge_t, weight_t> const &graph_view,


So, are we using the legacy graph for this? or are we using the legacy version for SG and the new version for MG?

seunghwak · 2021-03-25T22:24:49Z

cpp/tests/utilities/generate_graph_from_edgelist.cu

-  auto const col_comm_size = col_comm.get_size();
-
-  vertex_t number_of_vertices = static_cast<vertex_t>(vertices.size());
+template <typename vertex_t>


Do you still need the changes in this file after the test updates?

Not necessarily, I will verify this. It does successfully allow supporting of generating a multi-gpu graph without renumbering - in case we find another need for that.

That can be useful for debugging algorithms (knowing the numbering a priori and that it remains consistent across different shape MG tests).

I changed this file significantly in PR #1443, and this is not necessary for testing, we may better undo this change for now.

I will revert tomorrow.

seunghwak

Thanks!

rlratzel

One minor doc question and a couple of comments, but nothing to hold up my approval now.

rlratzel · 2021-03-26T21:30:04Z

cpp/include/algorithms.hpp

+ * @tparam     graph_t               Type of graph
+ *
+ * @param[in]  handle                Library handle (RAFT). If a communicator is set in the handle,
+ * @param[in]  graph                 input graph object


looks like the actual param name is graph_view, does that need to match (or should the param name be graph since it's not a view obj)?

It's a bit more pervasive. They are all graph views. In fact all of our algorithms are using graph views and calling them graphs.

I'm testing a change that changes louvain calls to consistently call it a graph_view. I believe we should do this with all algorithms (I'll try and make this change as I rework things to use the new graph primitives).

rlratzel · 2021-03-26T22:10:17Z

cpp/tests/community/mg_louvain_helper.hpp

@@ -0,0 +1,53 @@
+/*


Are mg_louvain_helper.[cu|hpp] really specific to louvain? I'm just wondering if they're generally useful, should they be renamed and placed in the cugraphtestutil lib?

They are currently specific to louvain. I think they might be more generally useful. But I wanted to have at least one more unit test that used it before moving into a more general location.

And FYI,

renumber_ext_vertices
https://github.com/rapidsai/cugraph/pull/1443/files#diff-4106792213a4762c10634a44ac8c92f194994ec0b595dba4eccaa822e8353978R215

unrenumber_local_int_vertices
https://github.com/rapidsai/cugraph/pull/1443/files#diff-4106792213a4762c10634a44ac8c92f194994ec0b595dba4eccaa822e8353978R245

and unrenumber_local_int_vertices
https://github.com/rapidsai/cugraph/pull/1443/files#diff-4106792213a4762c10634a44ac8c92f194994ec0b595dba4eccaa822e8353978R279

will be added once PR 1443 gets merged.

There will be some overlaps and some functions here may become obsolete.

rlratzel · 2021-03-26T22:12:06Z

cpp/tests/community/mg_louvain_test.cpp

+  size_t max_level;
+  double resolution;
+
+  // FIXME:  We really should have a Graph_Testparms_Base class or something


This is a good FIXME, I agree.

BradReesWork · 2021-03-30T14:18:48Z

@gpucibot merged

BradReesWork · 2021-03-30T14:19:53Z

@gpucibot merge

ChuckHastings added 5 commits February 22, 2021 19:19

Refactor Louvain with new graph primitives

97e7d83

1) Fix Dendrogram methods to be const consistent 2) Update experimental update_by_delta_modularity to use new graph primitives. Clean up unused local code

clean up include files, delete obsolete header

e1581d7

Merge branch 'branch-0.19' into fea_louvain_use_new_primitives

9b6fa0c

fix weight use in copy_v_transform_reduce_key_aggregated_out_nbr

11aeffa

use compute_out_weight_sums, add MG unit test

e8ab13a

seunghwak reviewed Mar 2, 2021

View reviewed changes

cpp/src/experimental/louvain.cuh Outdated Show resolved Hide resolved

rlratzel added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Mar 5, 2021

seunghwak mentioned this pull request Mar 8, 2021

Update graph partitioning scheme #1443

Merged

ChuckHastings added 2 commits March 15, 2021 13:14

Merge branch 'branch-0.19' into fea_louvain_use_new_primitives

f9fb0cd

MNMG Louvain working correctly

9e4421e

seunghwak reviewed Mar 16, 2021

View reviewed changes

ChuckHastings added 2 commits March 16, 2021 15:03

respond to PR comments

7cb4c13

rename louvain mg test file

d660d4c

change Testparams to Usecase

c90f608

ChuckHastings marked this pull request as ready for review March 25, 2021 13:52

ChuckHastings requested review from a team as code owners March 25, 2021 13:52

Merge branch 'branch-0.19' into fea_louvain_use_new_primitives

9bd82c2

ChuckHastings changed the title ~~[skip-ci] Update Louvain to use new graph primitives and pattern accelerators~~ Update Louvain to use new graph primitives and pattern accelerators Mar 25, 2021

BradReesWork requested review from aschaffer and afender March 25, 2021 14:20

BradReesWork added this to the 0.19 milestone Mar 25, 2021

afender reviewed Mar 25, 2021

View reviewed changes

aschaffer approved these changes Mar 25, 2021

View reviewed changes

seunghwak reviewed Mar 25, 2021

View reviewed changes

ChuckHastings added 5 commits March 26, 2021 13:44

clean up serial louvain tests

b9cdf40

Merge branch 'branch-0.19' into fea_louvain_use_new_primitives

ab2fb70

revert generate_graph_from_edgelist changes, not necessary with new test

b64e65b

update format and copyright information

bf522a4

missed a format issue

0810213

seunghwak approved these changes Mar 26, 2021

View reviewed changes

ChuckHastings added 2 commits March 26, 2021 17:26

add cuco/static_map guards for a few primitives that didn't have them

9e753ca

add guard for cuco static map usage in collect_values_for_keys

d2d1dfe

rlratzel approved these changes Mar 26, 2021

View reviewed changes

ChuckHastings added 2 commits March 26, 2021 18:48

update louvain parameters and documentation to reference a graph_view

4a26686

fix clang formatting issue

38dc0e5

afender approved these changes Mar 29, 2021

View reviewed changes

legacy louvain path now uses cuco also

a8c91c2

ChuckHastings mentioned this pull request Mar 29, 2021

[REVIEW] Create C++ documentation #1489

Merged

rapids-bot bot merged commit e60d9f7 into rapidsai:branch-0.19 Mar 30, 2021

ChuckHastings deleted the fea_louvain_use_new_primitives branch July 28, 2021 19:48

Update Louvain to use new graph primitives and pattern accelerators #1423

Update Louvain to use new graph primitives and pattern accelerators #1423

Conversation

ChuckHastings commented Feb 23, 2021 • edited Loading

seunghwak commented Mar 16, 2021

ChuckHastings commented Mar 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seunghwak commented Mar 16, 2021

ChuckHastings commented Mar 16, 2021

seunghwak commented Mar 16, 2021

ChuckHastings commented Mar 16, 2021

seunghwak commented Mar 16, 2021 • edited Loading

ChuckHastings commented Mar 16, 2021

seunghwak commented Mar 16, 2021

codecov-io commented Mar 25, 2021 • edited Loading

Codecov Report

afender left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afender Mar 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seunghwak left a comment

Choose a reason for hiding this comment

rlratzel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BradReesWork commented Mar 30, 2021

BradReesWork commented Mar 30, 2021

ChuckHastings commented Feb 23, 2021 •

edited

Loading

seunghwak commented Mar 16, 2021 •

edited

Loading

codecov-io commented Mar 25, 2021 •

edited

Loading

afender Mar 25, 2021 •

edited

Loading