Consider libp2p mono repo

[vasco-santos]

Taking into account the overhead that we experience in libp2p development when sometimes for getting a feature in we need to orchestrate several PRs just to land a change, we should consider turning libp2p into a mono repo. This would also help new contributors as they would not need to know where other modules are and that they would need to be updated (like the interop, in case its tests fail in CI).

Some of the repos that should be considered are the daemon, daemon client and libp2p-interfaces.

This issue should be used for this discussion, where we should put together benefits, drawbacks and lessons learned from js-ipfs change into a mono repo.

cc @jacobheun @achingbrain @hugomrdias

[jacobheun]

The other really nice thing we could do with this not listed is releasing preconfigured libp2p nodes more easily with every libp2p release so users don't have to configure anything if they just want something ready to go.

[achingbrain]

Some related modules have been grouped together as monorepos since this was opened:

• https://github.com/libp2p/js-libp2p-daemon
• https://github.com/libp2p/js-libp2p-interfaces
• https://github.com/libp2p/js-libp2p-peer-id
• https://github.com/libp2p/js-libp2p-webrtc-star

The primary piece of learning from js-ipfs is that choosing what to add to a monorepo is very important, and that you should not add everything unless you like very long build times.

Modules should also be homogeneous in terms of technology, scope and testing - if you have one GUI app in a monorepo and ten tiny utils, releasing changes to the utils takes as long as releasing changes to the GUI app which is a bit nuts.

Also flaky tests become that much painful.

Also, also, contributors tend to get scared off by the scope of a monorepo.

I don't think there are other modules that need grouping together in the libp2p ecosystem so I'm going to close this issue but it's always something worth keeping an eye on.

[p-shahi]

The subject of migrating js-libp2p components into a monorepo was brought up by @MarcoPolo and discussed in the 2023-03-07 js-libp2p triage meeting with @achingbrain @wemeetagain @maschad and @p-shahi

The reasons for moving to a monorepo given were:

• It is painful for maintainers to manage separate repos
• Even though its cleaner conceptually to have separate repos, they are harder to track.
  • In practice there's added overhead even with CI bots to make things smoother.

The original proposed solution was to move any potential component a user pulls into a simple js-libp2p app (i.e. TCP, mplex, Yamux, Noise, WebSocket, WebTransport, Interfaces, etc.) into a monorepo.
However, we decided not to do this because:

• Yamux, mplex, Noise, and GossipSub are maintained by ChainSafe and should be separate repos (to have other parties in ecosystem be responsible for libp2p components)
• A simple app or user may not use all transports or components (some transports like TCP are not used by browsers)

So instead of one monorepo, our current thought is to create different groupings of smaller monorepos. These would look like:

• js-libp2p: libp2p/interop, interfaces (consolidate interfaces to add and others to exclude)
• js-libp2p-transports: WebRTC, WebTransport, WebSocket
• js-libp2p-protocols: js-libp2p-kad-dht, identify, ping
• js-libp2p-peer-discovery: js-libp2p-mdns, js-libp2p-bootstrap

@MarcoPolo also brought up the idea of using Git subtrees (where one can make changes to a repo directly and from a monorepo pull those changes in or you can make changes from monorepo and push changes out) as an option - but no consensus here.

POC: https://github.com/marcopolo/js-libp2p-monorepo

[achingbrain]

Having spent more time that I'd like to admit pushing changes around during the IPFS Althing I am becoming more amenable to this as I'd like to spend more time shipping features and less time shipping dependency updates.

After [email protected] I'm going to set something up like this here:

• /packages/libp2p - the current repo
• /packages/interface - types from @libp2p/interface-libp2p - this will be the public API and will have no dependencies on anything we consider the internal API, the goal being this should not change often so projects like Helia can depend on it and be free of internal module churn
• /packages/interface-internal - a new module, @libp2p/interface-libp2p-internal that contains interfaces for internal components like ConnectionManager, TransportManager, etc. Modules in the services hash can depend on this to define the components they require to operate. By including this here it can be released alongside libp2p
• /packages/interop - this is for interop tests

Essentially the first bullet point from the comment above.

Docs, examples stay where they are.

[achingbrain]

Another problem that we have is releasing changes to modules ahead of libp2p.

For example the next libp2p release changes the peer store to be atomic, so that means:

1. Update @libp2p/interface-peer-store with the breaking changes and release a major
2. Update @libp2p/peer-store to implement the interface changes and release a major
3. Update consumers of the peer store @libp2p/bootstrap, @chainsafe/libp2p-gossipsub, etc and release majors
4. Update libp2p and release a major

Before 4 is completed there's a window during which js users see an update to a module, upgrade to it and then wonder why stuff doesn't work. Ts users will see a type error, though frequently they will ignore it and also wonder why stuff doesn't work.

We cannot control the release schedule of external modules (e.g. @chainsafe/libp2p-gossipsub) but being able to release 1-4 in one PR would go some way to reducing module churn and user surprise.

This adds a bit of weight to the whole-hog option of adding every supported libp2p module into the js-libp2p monorepo.

[p-shahi]

Merging interfaces into internal/interfaces and external/interfaces would be a breaking change. Maintainers have decided to wait on this for a bit, since 0.45.0 was a big breaking change release as well. We'll be doing this in the next month or so.

[p-shahi]

non-interface modules to monorepo added in #1828

[BigLep]

@achingbrain @p-shahi : A few things on this...

1. Since there is related work underway, I adjusted the status to be "In Progress". Please fix if that is wrong or if I oversteped.
2. Scope and tracking - have we defined what all is being moved in, where it's being moved, and what we expect of repos being moved in (e.g., archiving old repo, moving old issues, archiving repos, etc.)? I want to make sure we have a checklist that we're tracking against and that we're clear on the done criteria. These kinds of changes can be disruptive for consumers and we want to be clear about what we're doing and handle with care. There have been multiple instances of repo consolidation in the last year from PL EngRes, but one for potential reference is [Tracking issue] Release for the "over the hump" copy repo consolidation ipfs/boxo#202
3. Git commit history - This is follow on from chore: add non-interface modules to monorepo #1828 (comment) , but from an OSS perspective at the minimum let's keep the commit history. (Boxo effort recently did this and had some light tooling for it per https://github.com/ipfs/boxo/wiki/Copied-or-Migrated-Repos-FAQ#was-the-git-history-maintained-for-repos-that-were-copied-in).
   • Where we haven't done this, rather than force pushing, should we revert and then apply the commits?

[achingbrain]

1. 👍 🙏
2. All supported modules have been moved in. Archiving and issue triaging on the older repos is still in progress. With the exception of the consolidated interface modules everything is still published under the same module names as before (@libp2p/tcp, @libp2p/webrtc, etc) with no API changes so the hope is end-user disruption should be minimal.
3. The history has been restored so all good here I think.