[Feature] UUIDs, protocol versioning, v2 protocol w/ dag-cbor messagi…

…ng (#332)

* feat(net): initial dag-cbor protocol support

also added first roundtrip benchmark

* feat(requestid): use uuids for requestids

Ref: #278
Closes: #279
Closes: #281

* fix(requestmanager): make collect test requests with uuids sortable

* fix(requestid): print requestids as string uuids in logs

* fix(requestid): use string as base type for RequestId

* chore(requestid): wrap requestid string in a struct

* feat(libp2p): add v1.0.0 network compatibility

* chore(net): resolve most cbor + uuid merge problems

* feat(net): to/from ipld bindnode types, more cbor protoc improvements

* feat(net): introduce 2.0.0 protocol for dag-cbor

* fix(net): more bindnode dag-cbor protocol fixes

Not quite working yet, still need some upstream fixes and no extensions work
has been attempted yet.

* chore(metadata): convert metadata to bindnode

* chore(net,extensions): wire up IPLD extensions, expose as Node instead of []byte

* Extensions now working with new dag-cbor network protocol
* dag-cbor network protocol still not default, most tests are still exercising
  the existing v1 protocol
* Metadata now using bindnode instead of cbor-gen
* []byte for deferred extensions decoding is now replaced with datamodel.Node
  everywhere. Internal extensions now using some form of go-ipld-prime
	decode to convert them to local types (metadata using bindnode, others using
	direct inspection).
* V1 protocol also using dag-cbor decode of extensions data and exporting the
  bytes - this may be a breaking change for exising extensions - need to check
	whether this should be done differently. Maybe a try-decode and if it fails
	export a wrapped Bytes Node?

* fix(src): fix imports

* fix(mod): clean up go.mod

* fix(net): refactor message version format code to separate packages

* feat(net): activate v2 network as default

* fix(src): build error

* chore: remove GraphSyncMessage#Loggable

Ref: #332 (comment)

* chore: remove intermediate v1.1 pb protocol message type

v1.1.0 was introduced to start the transition to UUID RequestIDs. That
change has since been combined with the switch to DAG-CBOR messaging format
for a v2.0.0 protocol. Thus, this interim v1.1.0 format is no longer needed
and has not been used at all in a released version of go-graphsync.

Fixes: filecoin-project/lightning-planning#14

* fix: clarify comments re dag-cbor extension data

As per dission in #338, we are going
to be erroring on extension data that is not properly dag-cbor encoded from now
on

* feat: new LinkMetadata iface, integrate metadata into Response type (#342)

* feat(metadata): new LinkMetadata iface, integrate metadata into Response type

* LinkMetadata wrapper around existing metadata type to allow for easier
  backward-compat upgrade path
* integrate metadata directly into GraphSyncResponse type, moving it from an
  optional extension
* still deal with metadata as an extension for now—further work for v2 protocol
  will move it into the core message schema

Ref: #335

* feat(metadata): move metadata to core protocol, only use extension in v1 proto

* fix(metadata): bindnode expects Go enum strings to be at the type level

* fix(metadata): minor fixes, tidy up naming

* fix(metadata): make gofmt and staticcheck happy

* fix(metadata): docs and minor tweaks after review

Co-authored-by: Daniel Martí <[email protected]>

* fix: avoid double-encode for extension size estimation

Closes: filecoin-project/lightning-planning#15

* feat(requesttype): introduce RequestType enum to replace cancel&update bools (#352)

Closes: #345

* fix(metadata): extend round-trip tests to byte representation (#350)

* feat!(messagev2): tweak dag-cbor message schema (#354)

* feat!(messagev2): tweak dag-cbor message schema

For:

1. Efficiency: compacting the noisy structures into tuples representations and
   making top-level components of a message optional.
2. Migrations: providing a secondary mechanism to lean on for versioning if we
   want a gentler upgrade path than libp2p protocol versioning.

Closes: #351

* fix(messagev2): adjust schema per feedback

* feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID (#355)

* feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID

Closes: #349

* fixup! feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID

* fixup! feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID

when using error type T, use *T with As, rather than **T

* fixup! feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID

* fixup! feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID

Co-authored-by: Daniel Martí <[email protected]>

* feat: SendUpdates() API to send only extension data to via existing request

* fix(responsemanager): send update while completing

If request has finished selector traversal but is still sending blocks,
I think it should be possible to send updates. As a side effect, this
fixes our race.

Logically, this makes sense, cause our external indicator that we're
done (completed response listener) has not been called.

* fix(requestmanager): revert change to pointer type

* Refactor async loading for simplicity and correctness (#356)

* feat(reconciledloader): first working version of reconciled loader

* feat(traversalrecorder): add better recorder for traversals

* feat(reconciledloader): pipe reconciled loader through code

style(lint): fix static checks

* Update requestmanager/reconciledloader/injest.go

Co-authored-by: Rod Vagg <[email protected]>

* feat(reconciledloader): respond to PR comments

Co-authored-by: Rod Vagg <[email protected]>

* fix(requestmanager): update test for rebase

Co-authored-by: Daniel Martí <[email protected]>
Co-authored-by: hannahhoward <[email protected]>

benchmarks/testnet/virtual.go

@@ -16,6 +16,7 @@ import (
 	"google.golang.org/protobuf/proto"
 
 	gsmsg "github.com/ipfs/go-graphsync/message"
+	gsmsgv1 "github.com/ipfs/go-graphsync/message/v1"
 	gsnet "github.com/ipfs/go-graphsync/network"
 )
 
@@ -137,7 +138,7 @@ func (n *network) SendMessage(
 			rateLimiters[to] = rateLimiter
 		}
 
-		pbMsg, err := mes.ToProto()
+		pbMsg, err := gsmsgv1.NewMessageHandler().ToProto(peer.ID("foo"), mes)
 		if err != nil {
 			return err
 		}

cidset/cidset.go

@@ -4,32 +4,30 @@ import (
 	"errors"
 
 	"github.com/ipfs/go-cid"
+	"github.com/ipld/go-ipld-prime/datamodel"
 	"github.com/ipld/go-ipld-prime/fluent"
 	cidlink "github.com/ipld/go-ipld-prime/linking/cid"
 	"github.com/ipld/go-ipld-prime/node/basicnode"
-
-	"github.com/ipfs/go-graphsync/ipldutil"
 )
 
 // EncodeCidSet encodes a cid set into bytes for the do-no-send-cids extension
-func EncodeCidSet(cids *cid.Set) ([]byte, error) {
+func EncodeCidSet(cids *cid.Set) datamodel.Node {
 	list := fluent.MustBuildList(basicnode.Prototype.List, int64(cids.Len()), func(la fluent.ListAssembler) {
 		_ = cids.ForEach(func(c cid.Cid) error {
 			la.AssembleValue().AssignLink(cidlink.Link{Cid: c})
 			return nil
 		})
 	})
-	return ipldutil.EncodeNode(list)
+	return list
 }
 
 // DecodeCidSet decode a cid set from data for the do-no-send-cids extension
-func DecodeCidSet(data []byte) (*cid.Set, error) {
-	list, err := ipldutil.DecodeNode(data)
-	if err != nil {
-		return nil, err
+func DecodeCidSet(data datamodel.Node) (*cid.Set, error) {
+	if data.Kind() != datamodel.Kind_List {
+		return nil, errors.New("did not receive a list of CIDs")
 	}
 	set := cid.NewSet()
-	iter := list.ListIterator()
+	iter := data.ListIterator()
 	for !iter.Done() {
 		_, next, err := iter.Next()
 		if err != nil {

cidset/cidset_test.go

@@ -15,8 +15,7 @@ func TestDecodeEncodeCidSet(t *testing.T) {
 	for _, c := range cids {
 		set.Add(c)
 	}
-	encoded, err := EncodeCidSet(set)
-	require.NoError(t, err, "encode errored")
+	encoded := EncodeCidSet(set)
 	decodedCidSet, err := DecodeCidSet(encoded)
 	require.NoError(t, err, "decode errored")
 	require.Equal(t, decodedCidSet.Len(), set.Len())

dedupkey/dedupkey.go

@@ -1,27 +1,21 @@
 package dedupkey
 
 import (
+	"github.com/ipld/go-ipld-prime/datamodel"
 	"github.com/ipld/go-ipld-prime/node/basicnode"
-
-	"github.com/ipfs/go-graphsync/ipldutil"
 )
 
 // EncodeDedupKey returns encoded cbor data for string key
-func EncodeDedupKey(key string) ([]byte, error) {
+func EncodeDedupKey(key string) (datamodel.Node, error) {
 	nb := basicnode.Prototype.String.NewBuilder()
 	err := nb.AssignString(key)
 	if err != nil {
 		return nil, err
 	}
-	nd := nb.Build()
-	return ipldutil.EncodeNode(nd)
+	return nb.Build(), nil
 }
 
 // DecodeDedupKey returns a string key decoded from cbor data
-func DecodeDedupKey(data []byte) (string, error) {
-	nd, err := ipldutil.DecodeNode(data)
-	if err != nil {
-		return "", err
-	}
-	return nd.AsString()
+func DecodeDedupKey(data datamodel.Node) (string, error) {
+	return data.AsString()
 }

docs/processes.puml

@@ -12,16 +12,17 @@ partition "Top Level Interface" {
 if (operation type) then (outgoing request or incoming response)
 partition "Graphsync Requestor Implementation" {
 :RequestManager;
-if (operation type) then (incoming response)
-partition "Verifying Queries" {
+partition "Executing Requests" {
+:TaskQueue;
 fork
-:ipld.Traverse;
+:Executor;
 fork again
-:ipld.Traverse;
+:Executor;
 fork again
-:ipld.Traverse;
+:Executor;
 end fork
 }
+if (operation type) then (verified responses)
 partition "Collecting Responses" {
 fork
 :Response Collector;
@@ -33,21 +34,21 @@ end fork
 }
 :Responses returned to client;
 stop
-else (outgoing request)
+else (request messages)
 :Send Request To Network;
 endif
 }
 else (incoming request)
 partition "Graphsync Responder Implementation" {
 :ResponseManager;
 partition "Performing Queries" {
-:PeerTaskQueue;
+:TaskQueue;
 fork
-:ipld.Traverse;
+:QueryExecutor;
 fork again
-:ipld.Traverse;
+:QueryExecutor;
 fork again
-:ipld.Traverse;
+:QueryExecutor;
 end fork
 }
 }

docs/request-execution.puml

@@ -0,0 +1,102 @@
+@startuml Request Execution
+participant "GraphSync\nTop Level\nInterface" as TLI
+participant RequestManager
+participant TaskQueue
+participant RequestExecutor as RE
+participant ReconciledLoader
+participant TraversalRecord
+participant Verifier
+participant LocalStorage
+participant Traverser
+participant Network
+
+== Initialization ==
+
+TLI -> RequestManager ** : Setup
+TLI -> RE ** : Setup
+TLI -> TaskQueue ** : Setup
+
+== Executing A Request ==
+
+par
+note over TLI : Request Initiation
+TLI -> RequestManager : New Request
+RequestManager -> RequestManager : Create Request Context
+RequestManager -> TaskQueue : Push Request
+else
+note over RE: Request Execution
+TaskQueue -> RE : Next Request\nTo Process
+RE -> RequestManager : Initiate request execution
+RequestManager -> Traverser ** : Create to manage selector traversal
+RequestManager -> ReconciledLoader ** : create to manage 
+RequestManager -> RE : Traverser + ReconciledLoader
+note over RE: Local loading phase
+loop until traversal complete, request context cancelled, or missing block locally
+Traverser -> RE : Request to load blocks\nto perform traversal
+RE -> ReconciledLoader : Load next block
+ReconciledLoader -> LocalStorage : Load Block
+LocalStorage --> ReconciledLoader : Block or missing
+ReconciledLoader -> TraversalRecord : Record link traversal
+TraversalRecord --> ReconciledLoader
+ReconciledLoader --> RE : Block or missing
+opt block is present
+RE --> Traverser : Next block to load
+end
+end
+RE -> Network : Send Graphsync Request
+RE -> ReconciledLoader : remote online
+ReconciledLoader -> Verifier ** : Create new from traversal record
+ReconciledLoader -> RE
+note over RE: Remote loading phase
+loop until traversal complete, request context cancelled, or missing block locally
+Traverser -> RE : Request to load blocks\nto perform traversal
+RE -> ReconciledLoader : Load next block
+alt on missing path for remote
+ReconciledLoader -> LocalStorage : Load Block
+LocalStorage --> ReconciledLoader : Block or missing
+else
+loop until block loaded, missing, or error
+opt new remote responses
+  alt verification not done
+    ReconciledLoader -> Verifier : verify next response
+    alt success
+    Verifier --> ReconciledLoader : verified
+    ReconciledLoader -> ReconciledLoader : wait for more responses
+    else failure
+    Verifier --> ReconciledLoader : error
+    end
+  else verification done
+    alt next response matches current block load
+
+      alt next response contains a block
+        ReconciledLoader -> LocalStorage : store remote block
+        LocalStorage --> ReconciledLoader
+        ReconciledLoader -> ReconciledLoader : block laoded from remote
+      else next response does not contain block
+        opt next response is missing
+          ReconciledLoader -> ReconciledLoader : record missing path
+        end
+        ReconciledLoader -> LocalStorage : load block
+        LocalStorage --> ReconciledLoader : block or missing
+      end
+    else next response doesn not match
+      ReconciledLoader -> ReconciledLoader : error
+    end
+  end
+end
+opt remote goes offline
+ReconciledLoader -> LocalStorage : load block
+LocalStorage --> ReconciledLoader : block or missing
+end
+end
+ReconciledLoader -> TraversalRecord : Record link traversal
+TraversalRecord --> ReconciledLoader
+ReconciledLoader --> RE : Block, missing or error
+RE -> Traverser : Next block to load
+end
+end
+else
+Network -> RequestManager : New Responses
+RequestManager -> ReconciledLoader : Ingest Responses
+end
+@enduml