Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve memory footprint and performance of graph creation #3542

Conversation

VibhuJawa
Copy link
Member

@VibhuJawa VibhuJawa commented May 4, 2023

This PR improves the memory footprint in graph creation and resolves story: #3439

The metric we are concerned about is input_to_peak_ratio which is currently 12.1x. Ideally this should be in 2x range.

Metric Meaning: How much memory expands for an input graph of size x. So if you want to analyze a graph of size x, we need 12x GPU ram currently.

Memory overhead ratio reduction:

  • 12x to 6.14x for undirected (Reverse edges added)
  • 5.9x to 3.07x for directed graphs (No reverse edges)

Time Reduction:

  • 4.2x for scale 24
  • 1.67x for scale 23

image

Comparison Table

scale num_input_edges directed renumber input_memory_per_worker PR peak_allocation_across_workers Main: peak_allocation_across_workers Main: input_to_peak_ratio PR input_to_peak_ratio
23 134217728 TRUE TRUE 2.0GB 8.1GB 11.8GB 5.91 4.07
23 134217728 TRUE FALSE 2.0GB 6.1GB 11.8GB 5.91 3.07
23 134217728 FALSE TRUE 2.0GB 12.2GB 23.3GB 11.67 6.08
23 134217728 FALSE FALSE 2.0GB 12.2GB 23.3GB 11.67 6.08
24 268435456 TRUE TRUE 4.0GB 12.3GB 23.8GB 5.94 3.08
24 268435456 TRUE FALSE 4.0GB 12.3GB 23.8GB 5.94 3.08
24 268435456 FALSE TRUE 4.0GB 24.4GB 29.4GB 7.35 6.11
24 268435456 FALSE FALSE 4.0GB 24.4GB 29.4GB 7.35 6.11

Graph Creation:

Scale PR Time Main Time Speed Up
23 2.4 4.1 1.68
23 2.4 4.1 1.68
23 3.6 8.0 2.22
23 3.6 8.0 2.24
24 4.4 9.3 2.12
24 4.4 9.3 2.12
24 7.9 38.7 4.89
24 7.9 37.9 4.81
25 9.6 41.0 4.26
25 9.6 40.8 4.25
image

CC: @rlratzel

@VibhuJawa VibhuJawa added non-breaking Non-breaking change improvement Improvement / enhancement to an existing function labels May 4, 2023
@alexbarghi-nv
Copy link
Member

I think this PR may have some bugs, you may want to run the MG tests before finalizing it

@alexbarghi-nv
Copy link
Member

I ran into issues trying to use this with the latest changes to sampling and edge properties

@VibhuJawa VibhuJawa changed the title Improve memory footprint in graph creation Improve memory footprint and performance of graph creation May 12, 2023
@VibhuJawa VibhuJawa marked this pull request as ready for review May 16, 2023 06:30
@VibhuJawa VibhuJawa requested a review from a team as a code owner May 16, 2023 06:30
@VibhuJawa
Copy link
Member Author

VibhuJawa commented May 16, 2023

I think this PR may have some bugs, you may want to run the MG tests before finalizing it

@alexbarghi-nv , Can you try again. i think i have resolved all the bugs from my side and testing on multiple GPU configurations.

Copy link
Member

@alexbarghi-nv alexbarghi-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@alexbarghi-nv
Copy link
Member

/merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improvement / enhancement to an existing function non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants