-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
coreapi: DHT API #4804
coreapi: DHT API #4804
Conversation
6be1c4e
to
1516f2e
Compare
1516f2e
to
1d5d05f
Compare
#4807 should get merged first |
@magik6k is this ready for review? |
1d5d05f
to
d4b6cee
Compare
Rebased, should be ready for a review |
CI is red |
8aaf2e1
to
5f75628
Compare
I think there is a random test fail, will try to fix it first |
ae14365
to
77f6604
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any chance we could get the DHT commands converted over to use this API before merging? I understand if that would be too much too fast but this is a lot of unused code at the moment.
core/coreapi/dht.go
Outdated
dht, ok := api.node.Routing.(*ipdht.IpfsDHT) | ||
if !ok { | ||
return nil, ErrNotDHT | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just call FindPeer
directly on ipfs.node.Routing
? The interface requires that method.
core/coreapi/dht.go
Outdated
return | ||
} | ||
|
||
notif.PublishQueryEvent(ctx, ¬if.QueryEvent{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just call FindPeer
, get the result, return it to the user, and be done with it? We're throwing away all the other events anyways.
core/coreapi/dht.go
Outdated
return nil, err | ||
} | ||
|
||
dht, ok := api.node.Routing.(*ipdht.IpfsDHT) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here. We shouldn't be casting this. If we need some new functionality, we should add that to the interface.
core/coreapi/dht.go
Outdated
defer close(events) | ||
for p := range pchan { | ||
np := p | ||
notif.PublishQueryEvent(ctx, ¬if.QueryEvent{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again, maybe I'm missing something but this seems like pointless complexity. The caller of this method (FindProviders
) can't even use these events because we're registering for (and consuming) them.
core/coreapi/dht.go
Outdated
return errors.New("cannot provide in offline mode") | ||
} | ||
|
||
if len(api.node.PeerHost.Network().Conns()) == 0 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oof. This is going to hurt. It's fine to do this from the command as we amortize over the request and multiple arguments but listing all the connections just to see if we have some connections is going to be very expensive. Ideally, the DHT would detect this and return an error. Does it not do that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was it the dht command.
Looking at provide code in kad-dht, it invokes dht.GetClosestPeers
, which I think will error in that case (correct me if I'm wrong) -> https://github.com/libp2p/go-libp2p-kad-dht/blob/2bab49ef85b2d8547e95ca2a699b5be562cc78b6/lookup.go#L60
core/coreapi/dht.go
Outdated
|
||
//defer close(events) | ||
if settings.Recursive { | ||
err = provideKeysRec(ctx, api.node.Routing, api.node.DAG, []*cid.Cid{c}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm pretty sure this needs to be an offline DAG.
core/coreapi/coreapi.go
Outdated
@@ -59,6 +59,11 @@ func (api *CoreAPI) Pin() coreiface.PinAPI { | |||
return (*PinAPI)(api) | |||
} | |||
|
|||
// Dht returns the DhtAPI interface implementation backed by the go-ipfs node | |||
func (api *CoreAPI) Dht() coreiface.DhtAPI { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
High level comment: I'd rather not have a DHT API. The DHT is just a backend for other stuff (the routing interface). While I know we currently have a command, I'd prefer not to bake this into the CoreAPI. Is there a design discussion for the CoreAPI anywhere?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a design discussion for the CoreAPI anywhere?
I'm trying to follow the js coreapi spec in most places - for DHT https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/DHT.md.
I did only include the basic functions which may be useful in some cases.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
I've submitted an issue to discuss this (https://github.com/ipfs/interface-ipfs-core/issues/249). We can probably move forward discuss this in parallel. It's not the end of the world, just a bit unfortunate.
core/coreapi/dht.go
Outdated
} | ||
|
||
func provideKeysRec(ctx context.Context, r routing.IpfsRouting, dserv ipld.DAGService, cids []*cid.Cid) error { | ||
provided := cid.NewSet() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just a copy of the existing functionality but... it's going to hurt regardless. Could we get a TODO: Use a bloom filter?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
core/coreapi/dht.go
Outdated
for _, c := range cids { | ||
kset := cid.NewSet() | ||
|
||
err := dag.EnumerateChildrenAsync(ctx, dag.GetLinksDirect(dserv), c, kset.Visit) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same, could we get a "TODO: provide and enumerate in parallel with a small buffer".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will get that with #4333
core/coreapi/interface/dht.go
Outdated
) | ||
|
||
// DhtAPI specifies the interface to the DHT | ||
type DhtAPI interface { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rather use the interface defined in https://github.com/libp2p/go-libp2p-routing/ (although we'd probably want to change it to return channels). Alternatively, we could do this for now with a big "we're going to change this" todo.
36dec57
to
6665373
Compare
Thanks for the review and pointers.
I guess it's probably better to get this done sooner than later. This is going to be at least a slightly breaking change. Currently when you run some dht command with |
6665373
to
e326b7f
Compare
49033ec
to
c0e2f54
Compare
(should be ready to merge, though we might want one more review) |
5b87efc
to
c7ad317
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two nits but nothing that we can't fix later (well, except the With
prefix but I don't feel strongly about that).
core/coreapi/dht.go
Outdated
errCh := make(chan error) | ||
go func() { | ||
for _, c := range cids { | ||
dserv := dag.NewDAGService(blockservice.New(bs, offline.Exchange(bs))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's construct this once.
|
||
// WithRecursive is an option for Dht.Provide which specifies whether to provide | ||
// the given path recursively | ||
func (dhtOpts) WithRecursive(recursive bool) DhtProvideOption { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the rational behind the With
prefix?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Options before refractor used it, this didn't get updated with rebase. Will fix
Hm. Actually, is there any way we could get some better test coverage or will that come when we switch commands over to using this interface? |
15fcac3
to
c1c8b4d
Compare
coverage++ (73% on coreapi/dht.go) (Circle appears to be borked/annoyingly slow - #5389 may or may not be related) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SGWM
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
License: MIT Signed-off-by: Steven Allen <[email protected]>
License: MIT Signed-off-by: Łukasz Magiera <[email protected]>
c1c8b4d
to
86f9eb7
Compare
Based on #4802
Replaces #4774
TODOs: