Consolidate libp2p abstractions and core types

[raulk]

... into go-libp2p-core! 🎉

We have positive consensus about consolidating the abstractions and core types of go-libp2p in a single repository. The rationale is clear:

• Reduces cognitive load when reasoning about the system.
• Makes go-libp2p significantly more approachable for newcomers and contributors.
• Decreases repo count by around 20%, thus alleviating maintenance burden and unnecessary complexity.
• Offers an opportunity to exterminate inconsistently named repos, and to fix package naming snafu.
• Enables building alternative go-libp2p backends easier by targeting a single collection of abstractions (e.g. lightweight proxy that binds to a remote/co-local libp2p daemon).

Alternatives considered

There were two reasonable approaches, with their pros, cons, and notes:

1. Pull the abstractions and core types into go-libp2p under a dedicated package: skel (skeleton), abstract, or similar. Move out the constructor, host implementations and protocols.

   • pro: putting the skeleton inside the entrypoint repo (go-libp2p) reduces indirection for beginners, when learning about the system by browsing code.
   • con: stops being the development entrypoint; the constructor living elsewhere adds an indirection when hacking together a libp2p app.
   • con: landslide change.
   • note: doing this progressively (absorbing abstractions, then moving out hosts and constructor) was impossible thanks to gomod's affinity for VCS; it was rather all-or-nothing, as gomod's approach does not play well with module cross-references.
     • e.g. go-libp2p hosts reference go-libp2p-interface-connmgr which aliases back to go-libp2p.
     • Locking in cross-references via commit hashes or tags is a catch-22.

2. Place abstractions in a new module, e.g. go-libp2p-core.

   • pro: constructor stays in go-libp2p, thus that module continues serving as a true entrypoint for development.
   • con: new users need to discover that the conceptual model of the system lives in another repo.

Selected approach

I started with option 1, and after consultation with @Stebalien, decided to migrate to option 2. Hence, option 2 is what I'm proposing to adopt.

Here is a diagram of the proposed refactor, to ease digestion:

WARNING: This diagram is now outdated, but it was accurate when the PR was filed.

Proposed changes

All of these PRs are work in progress (despite the PRs not signalling so) and subject to change based on feedback.

This is a sizeable architectural change that will bring long-awaited improvements to our development workflow and experience, but we have to get it right. Consensus is the decision process I am invoking.

** recipients:**

go-libp2p-core: libp2p/go-libp2p-core#1
go-libp2p-testing: libp2p/go-libp2p-testing#1

donors:

libp2p/go-conn-security#12
libp2p/go-flow-metrics#6
libp2p/go-libp2p-discovery#24
libp2p/go-libp2p-host#29
libp2p/go-libp2p-interface-connmgr#16
libp2p/go-libp2p-interface-pnet#12
libp2p/go-libp2p-net#44
libp2p/go-libp2p-peer#48
libp2p/go-libp2p-peerstore#69
libp2p/go-libp2p-pnet#27
libp2p/go-libp2p-protocol#8
libp2p/go-libp2p-routing#43
libp2p/go-libp2p-transport#49
libp2p/go-stream-muxer#28
libp2p/go-libp2p-crypto#60
libp2p/go-testutil#24 (test helpers and suites)

modules requiring backwards-compatible changes:

#601
libp2p/go-libp2p-blankhost#20
multiformats/go-multistream#37

Summary of change in these modules: The Host interface was referencing the go-multistream.MultistreamMuxer struct directly. I extracted an interface protocol.Switch out of it, which in turn embeds protocol.Router (methods to attach handlers) and a protocol.Negotiator (methods to negotiate protocols).

Host now references the new interface, and backwards-compatible changes were required in those modules.

Testing the changes for backwards compatibility

Unfortunately CI won't work because the go-libp2p-peerstore dependency is affected by this roadblock in the go toolchain: golang/go#31491

To test building these changes locally:

$ git clone --recursive [email protected]:libp2p/workspace-go-libp2p.git
$ cd workspace-go-libp2p
$ # add the multiformats/go-multistream 
$ git submodule add [email protected]:multiformats/go-multistream.git
$ # check out all consolidate-skeleton branches
$ git submodule foreach 'git checkout consolidate-skeleton || true'
$ # make modules cross-reference each other locally
$ ./workspace.sh local
$ # workaround to fix the import path of go-multistream in the replace directives
$ git submodule foreach 'go mod edit -replace=github.com/multiformats/go-multistream=../go-multistream -dropreplace=github.com/libp2p/go-multistream'
$ git submodule foreach 'go build ./...'

(TODO: run all tests)

Notes

(will keep enhancing this section with notes that come off my mind)

• In deprecated donors, I have used moved as a package alias and deprecated.go as a filename. No strong opinion here, but I think it intuitively screams the signal that you should not use this.
• In order to test for backwards compatibility and non-breakage with high fidelity, I have intentionally left untouched everything in go-libp2p that was not strictly necessary to change.
  • All implementations are therefore referencing deprecated types.
  • Once we're confident we're fully backwards compatible, we should practice what we preach and switch to go-libp2p-core everywhere.
• I have not reviewed any docs during this process, I just copy pasted. We should take this opportunity to do so.
• I don't want to hold off on merging too long, as this has been a significant undertaking, a big code coordination effort, and I'm afraid branches will diverge rapidly, code will slide, and it'll be a mess.

TODO

Open points:

• what do we do with go-libp2p-crypto? (decision: absorb)
• alias central types at root of go-libp2p-core.
• what do we do with go-libp2p-loggables? (decision: nothing)
• what do we do with global variables? They cannot be aliased, as they are assigned by value. Keeping them in the original packages is misleading, and removing them introduces breakage.
• what do we do with helper functions?
• review all docs (@yusefnapora?).
• review open issues and PRs in donor repos, and close or move over to core.

[hsanjuan]

When this lands, can we have a small summary of breaking changes or upgrade steps? It should be fine to break things now that everything is go-modded. It's worse that people stay depending on deprecated repos just because they work and missed this consolidation.

[yusefnapora]

I think this is a great idea, and it will make it so much simpler to approach go-libp2p and explain the core functionality.

I'll start going through tomorrow and see if there's anything useful from the donor READMEs that didn't carry over, and also make sure the godoc comments still make sense.

I'm also going to need you to tell me what you used to make that sweet diagram...

[vyzo]

We have a problem the way the function aliases are implemented:
The current set of patches moves the function by creating a variable that points to the new function.
This incurs a performance penalty as:

• direct calls are turned into indirect calls that have to go through the heap
• compiler optimizations such as inlining and escape analysis are inhibited.

I propose to do the move of functions by preserving the old functions and changing the body to simply call the new core function. This will allow the compiler to inline and incur 0 performance penalty.

[vyzo]

So to be clear, the aliasing of function should be done like this:

func OldModuleFunc(args ...) {
  return core.NewModuleFunc(args ...)
}

[raulk]

@vyzo +1 makes sense. Definitely more compiler optimisation-friendly.

Just to clarify, we'll be moving go-libp2p-* away from deprecated packages onto core. I'll do that after these PRs are merged, because:

1. Keeping implementations pointing to deprecated packages helps us prove we're backwards-compatible.
2. I want to get buy-in to lock in these changes before doing the big bang of PRs across the entire codebase.

[vyzo]

Yeah, let's do it gradually. As long as there is no performance regression, there is no impetus for immediate big bang.

[raulk]

@vyzo I've now updated all donor modules to alias deprecated package funcs by proxying the call over to go-libp2p-core.

[Stebalien]

I'd also like to consider moving some of the core types to the root package. While it'll clutter the namespace, it may make it easier for users (one import).

Core types include:

• Host, Conn, Stream, (Network?)
• peer.ID -> PeerID
• protocol.ID -> ProtocolID
• PeerInfo
• Keys?
• alias Multiaddr?

---

crypto

IMO, we might want to include that as well.

loggables

Don't include them. We're planning on switching to a third-party event logging framework anyways.

---

For completeness, other paths considered includes:

• Put the interfaces in go-libp2p and create a go-libp2p-default (or something like that) with the actual implementation. I actually kind of like this approach as library authors would then import go-libp2p instead of go-libp2p core (for the interfaces) but it would be a widely breaking change.

[raulk]

@Stebalien

I'd also like to consider moving some of the core types to the root package. While it'll clutter the namespace, it may make it easier for users (one import).

Hm, not sure about this. Moving just some types creates irregularity. When finding a type, developers would have to think: "is this type core enough to be at the root?" Actually, the question is different: "did the maintainers of go-libp2p consider this type core enough at the time to put it at the root?"

Unless we can all agree on a heuristic that's so natural and obvious that it's intuitive (not just to us, but to newcomers who are clueless about libp2p), I'm more inclined to adopt a clean-cut, normalised structure.

That said, we can introduce type aliases at the root for better ergonomics.

[Stebalien]

That said, we can introduce type aliases at the root for better ergonomics.

Let's try that. My main concern is that there's nothing at the root so it's kind of hard to explore the API.

[raulk]

We appear to have positive consensus on the change. Merge bus shall depart soon! 🚌 Please get your approvals or final judgement calls flowing!

Since this change comprises ~20 PRs, let's consolidate review approval on the go-libp2p-core PR: libp2p/go-libp2p-core#1

---

I've tested the changes against go-ipfs (@Stebalien) and ipfs-cluster (@hsanjuan) with no changes, but with replace directives to map libp2p onto the refactored workspace.

Compilation succeeded, and so did a runtime smoke test (ipfs daemon).

It appears that backwards-compatibility is preserved. If we discover the opposite at a later time, our usage of gomod has us covered.

[ghost]

Compilation succeeded, and so did a runtime smoke test (ipfs daemon).

How about a make test? :)

[raulk]

The refactor is now complete! 🎉 All affected modules have been released with minor version bumps.

Check out the new core abstractions at: https://github.com/libp2p/go-libp2p-core/ It includes shiny new godocs for all types as well ;-)