Add a TraversingLinksystem supporting traversal resumption #354
Conversation
There was a problem hiding this comment.
What is the cost of resuming a traversal? As a user, that would be my main worry. For example, if I have a deeply nested graph, and I want to resume at a deep point, will the resumption need to do work to get back there? One can imagine a similar situation with very wide nodes, such as a very long list where I want to resume a traversal towards the end of the list.
traversal/traversing_linksystem.go
Outdated
| position: 0, | ||
| pathOrder: make(map[int]datamodel.Path), | ||
| pathTree: newPath(nil), | ||
| target: nil, |
There was a problem hiding this comment.
nit: you can omit zero values from composite literals
traversal/traversing_linksystem.go
Outdated
| return []datamodel.Link{pn.link} | ||
| } | ||
| return []datamodel.Link{} | ||
| } else if len(segs) == 1 { |
traversal/traversing_linksystem.go
Outdated
| next := segs[0] | ||
| if child, ok := pn.children[next]; ok { | ||
| return child.allLinks() | ||
| } else if !ok { |
There was a problem hiding this comment.
nit: this can just be an else
Instead of keep track of all previous links that have been followed, which is already done in a progress, you are now keeping track of the ipld.PathSegment map as well. |
|
I think my main concern/ask for either this or #358 isn't even so much the code at this point -- it's: can we get a serial fixture to appear in ipld/ipld with it? I realize some of this may feel like it's golang specific APIs, but I doubt it's going to stay that way -- other languages will need to replicate the behavior at some point, and we might as well be ready for them with fixtures. Also, personally, I'm finding serial fixtures a lot easier to review for this stuff. There's a |
|
I would like to leave this as a golang specific API at this point, and not attempt to define a specification for how resumption is exposed that is cross-language standardized until we understand what is appropriate in other language. It feels generally like we are not proposing a change in spec or standard at this point, but rather providing a language-specific convenience method. |
Co-authored-by: Hannah Howard <[email protected]>
|
@willscott regarding the cost - sounds good. Perhaps it would be good to summarize this in the docs; as someone reading the API docs and not knowing how traversals and selectors are implemented, it would be very hard to estimate how expensive a resumption could be. |
|
@willscott : is this still needed in light of #358? If so, are you able to incorporate @mvdan 's feedback? |
|
This is fairly separable, and I plan to close this PR and move this code with the go-car PR that makes use of it. It's probably not a generic interface we want to have to keep supporting forever at this library level, but it was useful for a couple related PRs that have been set up already using the same structure. |
|
@willscott : this came up during 2022-04-05 triage. Are you able to close this now? |
|
Closing as this logic has now been fully transferred to ipld/go-car#291 |
This is a proposal for how we would extend the current Progress / traversal structure to optionally support efficient resumption of traversals.
The trade-off is that more state is retained in order to know which links have been seen by which points in the traversal. This proposal handles that decision by allowing the user to wrap their
ProgressviaWithTraversingLinksystem. This method wraps the linksystem block loader in order to track and appropriately skip forward when a subsequentresumeris used.I'd be happy if we could arrive at a more intuitive design, but this seems like the path of least resistance for maintaining current API compatibility.