Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce peak memory requirement in graph creation (part 1/2) #2070

Merged
merged 13 commits into from
Feb 22, 2022

Conversation

seunghwak
Copy link
Contributor

@seunghwak seunghwak commented Feb 13, 2022

Previously, graph creation requires roughly edge data size * 2 + alpha (if we can destroy the input edge data and reclaim the memory, * 3 if the input edge data cannot be destroyed). This PR cuts this to edge data size * 1.5 + alpha.

This is a breaking PR for projects using functions under https://github.com/rapidsai/cugraph/blob/branch-22.04/cpp/include/cugraph/utilities/shuffle_comm.cuh (@aschaffer You may be affected).

There will be a follow-up PR further optimizing memory footprint. Wrapping up this PR to avoid creating a giant PR.

@seunghwak seunghwak added 3 - Ready for Review improvement Improvement / enhancement to an existing function breaking Breaking change labels Feb 13, 2022
@seunghwak seunghwak added this to the 22.04 milestone Feb 13, 2022
@seunghwak seunghwak requested a review from a team as a code owner February 13, 2022 08:23
@seunghwak seunghwak self-assigned this Feb 13, 2022
…dgelist to pass mem_frugal=true to limit the maximum allocation chunk size to avoid malloc failure due to fragmentation with the pool allocator
cpp/src/detail/shuffle_wrappers.cu Outdated Show resolved Hide resolved
cpp/tests/utilities/test_graphs.hpp Outdated Show resolved Hide resolved
@seunghwak seunghwak changed the title Reduce peak memory requirement in graph creation Reduce peak memory requirement in graph creation (part 1/2) Feb 16, 2022
@codecov-commenter
Copy link

codecov-commenter commented Feb 19, 2022

Codecov Report

Merging #2070 (edd7cdd) into branch-22.04 (f6a92fc) will increase coverage by 0.09%.
The diff coverage is n/a.

Impacted file tree graph

@@               Coverage Diff                @@
##           branch-22.04    #2070      +/-   ##
================================================
+ Coverage         73.54%   73.64%   +0.09%     
================================================
  Files               156      156              
  Lines             10316    10332      +16     
================================================
+ Hits               7587     7609      +22     
+ Misses             2729     2723       -6     
Impacted Files Coverage Δ
python/cugraph/cugraph/structure/number_map.py 67.02% <0.00%> (ø)
python/cugraph/cugraph/linear_assignment/lap.py 71.42% <0.00%> (ø)
python/cugraph/cugraph/link_prediction/sorensen.py 85.18% <0.00%> (ø)
...graph/cugraph/centrality/betweenness_centrality.py 89.65% <0.00%> (ø)
...raph/structure/graph_implementation/simpleGraph.py 74.30% <0.00%> (ø)
.../cugraph/tests/test_edge_betweenness_centrality.py 84.93% <0.00%> (+0.60%) ⬆️
...graph/cugraph/tests/test_betweenness_centrality.py 83.01% <0.00%> (+0.62%) ⬆️
python/cugraph/cugraph/tests/test_graph_store.py 100.00% <0.00%> (+1.56%) ⬆️
python/cugraph/cugraph/utilities/utils.py 72.92% <0.00%> (+2.62%) ⬆️
python/cugraph/cugraph/tests/test_utils.py 75.55% <0.00%> (+4.44%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6a92fc...edd7cdd. Read the comment docs.

@BradReesWork
Copy link
Member

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 4679a28 into rapidsai:branch-22.04 Feb 22, 2022
@seunghwak seunghwak deleted the enh_mem_footprint branch August 11, 2022 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking Breaking change improvement Improvement / enhancement to an existing function
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants