Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add delete subgraph functionality #1053

Merged
merged 3 commits into from
Sep 28, 2023

Conversation

zekemorton
Copy link
Collaborator

@zekemorton zekemorton commented Jul 27, 2023

This PR adds delete subgraph functionality to the jgf reader and resource query. This is a part of the elasticity effort and allows removal of a piece of the resource graph. This function works by passing in a path of a node in the graph (ex /medium0/rack3) and will remove the tree with that node as root (ex /medium0/rack3/*).

This implementation requires two main steps. The first step is to identify all boost graph nodes in the subgraph. This is done via a DFS on the out edges of each node. Note that one of the out edges of a node is its parent, so in this case a string comparison of the path was used to determine that an out edge is not the parent. The second step is to remove the nodes from the graph. Because we are using listS instead of vecS, actually deleting the nodes would invalidate the vertex descriptors, which would invalidate the resource graph metadata used for vertex descriptor look ups. Instead, this PR clears all edges from "deleted nodes" and removes them from the resource graph metadata. This makes them unaccessible from traversals from the main graph and unaccessible from resource graph metadata look ups.

This PR also adds this functionality to resource query and adds new tests for these features.

WIP to do items:

  • Add automated tests for remove subgraph

@milroy milroy self-requested a review July 27, 2023 20:11
@milroy milroy self-assigned this Jul 27, 2023
@zekemorton zekemorton force-pushed the delete-node2 branch 6 times, most recently from 99a505e to e66b611 Compare August 1, 2023 18:01
@zekemorton zekemorton changed the title add delete subgraph functionality [WIP] add delete subgraph functionality Aug 1, 2023
Copy link
Member

@milroy milroy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking very good. After you fix a few nits and typos and address one question this should be good to merge. Thanks a lot!

resource/readers/resource_reader_jgf.cpp Outdated Show resolved Hide resolved
resource/readers/resource_reader_base.hpp Show resolved Hide resolved
resource/readers/resource_reader_jgf.cpp Outdated Show resolved Hide resolved
resource/readers/resource_reader_jgf.cpp Outdated Show resolved Hide resolved
resource/readers/resource_reader_jgf.cpp Outdated Show resolved Hide resolved
@zekemorton
Copy link
Collaborator Author

After finding a segfault centered around not deleting elements from the by_outedge member of the resource graph metadata, I added in a few new changes to solve this issue.

  • remove the by_outedge for each vertex that we remove from the resource graph
  • remove any edges from the vertex who is parent to the root of the removed subgraph to the root vertex (note that there can be more than one)
  • add in tests that try to fully allocate the resource graph after removing a resource

@jameshcorbett
Copy link
Member

After finding a segfault

Pincer maneuver complete

@zekemorton
Copy link
Collaborator Author

@milroy I add the most recent changes requested including the two TODO messages and spacing fixes.
I am not sure what the issue is with the python version tests though. Any thoughts there?

@milroy
Copy link
Member

milroy commented Sep 27, 2023

Looks like you figured out the runner issue with PR #1080. I'm approving this PR, but let's wait to merge it until #1080 is merged.

milroy
milroy previously approved these changes Sep 27, 2023
Copy link
Member

@milroy milroy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes! LGTM.

@milroy
Copy link
Member

milroy commented Sep 28, 2023

@zekemorton since PR #1080 is merged you can rebase this PR and set MWP.

Problem: adding elasicity into fluxion requires the addition of a
new feature to remove subgraphs from the resource graph.

This implementation removes nodes from the resource graph metadata
and clears the node of edges. This makes those nodes essentially
unreachable through traversal or look up from the metadata. Actually
deleting nodes from the resource graph would invalidate the
vertex descriptors and resource graph metadata. This is because
the resource graph is using a vecS instead of a listS for the Boost
adjacency list.
Problem: the remove subgraph feature is a part of the readers but
does not have functionality in resource query

Add support for remove graph in resource query
problem: the remove subgrapgh feature added new code that
needs to be tested as a part of automated ci

add tests for this new feature
@mergify mergify bot dismissed milroy’s stale review September 28, 2023 16:25

Approving reviews have been dismissed because this pull request
was updated.

@zekemorton zekemorton added the merge-when-passing mergify.io - merge PR automatically once CI passes label Sep 28, 2023
@codecov
Copy link

codecov bot commented Sep 28, 2023

Codecov Report

Merging #1053 (93f57cc) into master (4b16578) will increase coverage by 0.1%.
The diff coverage is 83.9%.

❗ Current head 93f57cc differs from pull request most recent head e9c4bfc. Consider uploading reports for the commit e9c4bfc to get more accurate results

@@           Coverage Diff            @@
##           master   #1053     +/-   ##
========================================
+ Coverage    71.6%   71.7%   +0.1%     
========================================
  Files          89      89             
  Lines       11569   11665     +96     
========================================
+ Hits         8289    8371     +82     
- Misses       3280    3294     +14     
Files Coverage Δ
resource/readers/resource_reader_hwloc.hpp 100.0% <ø> (ø)
resource/readers/resource_reader_jgf.hpp 100.0% <ø> (ø)
resource/readers/resource_reader_rv1exec.hpp 100.0% <ø> (ø)
resource/readers/resource_reader_jgf.cpp 69.4% <98.6%> (+3.2%) ⬆️
resource/readers/resource_reader_grug.cpp 83.3% <0.0%> (-1.1%) ⬇️
resource/readers/resource_reader_rv1exec.cpp 73.0% <25.0%> (-0.7%) ⬇️
resource/readers/resource_reader_hwloc.cpp 64.8% <0.0%> (-0.8%) ⬇️
resource/utilities/command.cpp 75.4% <68.4%> (-0.2%) ⬇️

@zekemorton
Copy link
Collaborator Author

Approving reviews have been dismissed because this pull request was updated.

@milroy looks like it might need another approval since doing the rebase

Copy link
Member

@milroy milroy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving again in the hopes that this will trigger MWP action.

@mergify mergify bot merged commit 4e3afdd into flux-framework:master Sep 28, 2023
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merge-when-passing mergify.io - merge PR automatically once CI passes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants