Making js-waku adaptive and modular (+ ESM migration)

[D4nte]

Problem

• Several libp2p libraries have migrated to ESM only
• The recommendation from the ecosystem is to convert one's package to ESM, the alternative is to use dynamic imports.
• Better protobuf library protons is ESM only, and generate code that expect to be used in an ESM package, meaning that if one were to use dynamic imports, they would need to post-process code generated by protons.

It would also be possible to not migrate to ESM and:

1. Use dynamic import for libp2p dependencies
2. Continue to use ts-proto instead of protons.

However, we have already identified reasons to move away from ts-proto: #335

Also, the dynamic import solution is cumbersome to use as imports needs to be wrapped in async functions, making the code less maintainable.

Finally, ESM is the only module management system adopted by browsers, which are the target audience for js-waku.

Solution

Migrate js-waku to ESM.

Limitations

Only modern browsers support ESM

js-waku's usage is already limited to modern browser due to the usage of BigInt operations (**, cannot be polyfiled) by @noble dependencies (cryptographic libraries). We recently migrated to said dependencies as they are better for the browser: #599.

Only latest NodeJS version (>16) support ESM

js-waku is already limited to running in NodeJS >16.

Tooling support of ESM is experimental

This is correct. However, as the ecosystem is moving to ESM then we can expect the tooling to get better.
Also note that the alternative of not migrating to ESM in the first section.

Other Benefits

Migration from Webpack to Rollup

As we need to re-configure bundling tools for "type": "module", we will take this opportunity to move from Webpack to Rollup.
Rollup is easier to use than Webpack, with zero-conf plugins to handle bundling to several formats, polyfilling, etc.

Make js-waku adaptive and modular

The current js-waku codebase is not as modular as it could be:

• It is not possible to disable Waku Relay Allow disabling of Relay #784
• It is unclear if the Waku Message Version 1 module is tree-shaked
• It is unclear if ENR & DNS Discovery implementations are tree-shaked (most likely note due to the if then statement used for bootstrapping).

The js-libp2p libraries have gone through a massive refactor:

• Most libraries are now written in TypeScript
• Most libraries are now ESM only
• Clear separation of types (@libp2p/interface-*) and function (@libp2p/*) has been made
• Improved code practice (likely due to usage of TypeScript), means better modularity in the way libp2p is configured (instances instead of classes being passed, this helps a lot for the pubsub config).

Note that we have a number of new cryptography-based protocols to be implemented in js-waku:

• Waku Message version 2 with noise and noise sessions: https://rfc.vac.dev/spec/35/ https://rfc.vac.dev/spec/37/
• Waku Relay RLN https://rfc.vac.dev/spec/17/

These protocols will come with their own set of new dependencies, possibly using WASM code.
They may introduce incompatibilities with some browser or NodeJS version, or polyfilling.
They may possibly need custom building and bundling.
Maybe some of the new crypto dependencies will have no/poor tree-shakeable support, etc.

Ideally these implementations should live in their own repo/package, separated from js-waku core library.

For this to happen, we need a proven way to enable a modular usage of protocol a la js-libp2p, by having current available protocols already separated from the core js-waku package.

However, we want Waku to remain an easy to use, with a great out of the box experience, and it is why we will aim to maintain good defaults while providing increased modularity.

Also note, it is not possible to use js-waku in React Native due to @noble/secp256k1.
This library is only used by ENR and Waku Message version 1.
One should be able to easily use js-waku without these protocols or with custom React Native implementations of them.
An more modular architecture will also enable this use case.

Draft Roadmap

• Finish migration to ESM
• Start migrating to a more flexible way of creating a Waku node with selection of protocols, while keeping good default and simple API
• move js-waku to @waku/core, publish first version of @waku/core
• Properly implement @libp2p/interface-peer-discovery for js-waku/boostrap, DNS Discovery, extract it from @waku/core
• Redefine API for Waku message exchange protocols (store, filter, lightpush, relay) to modularize the usage of Waku Message version 1, extract version 1 to separate package
• Implement Waku noise, Waku RLN Relay
• Continue to extract Waku protocols in separate packages to improve modularity.

We might be able to reach a stage where @waku/core only provides good defaults but anyone could use Waku Relay, Store, Message, js-implementations directly with libp2p by passing them to createLibp2p.

Definition of Done

JS-Waku is "type": "module" and usable:

• With web framework that uses a package manager such as React
• With NodeJS chore(deps): bump js-waku to 0.25.0 waku-tests#4
• With html <script> tag

Tasks

• Update libp2p deps (see next item) Upgrade to latest libp2p libraries #803
• Fix bundle build to use in <script> tag: js-libp2p cannot be bundled with Rollup due to the usage of node's cluster, this is fixed by the latest js-libp2p release Upgrade to latest libp2p libraries #803
• publish to @waku/core
• Remove dupe CreateOptions interfaces Remove dupe CreateOptions interfaces #1000
• See if producing a CJS build is possible and useful. CJS is only useful for older NodeJS versions that we already do not support.
• Update ReactJS, JavaScript/HTML examples to @waku/core, introduce a NodeJS example (see status-im/waku-tests).
  • Ensure it works (Invalid exports field in the package.json #770)
  • Do the necessary or identify necessary steps to avoid touching ReactJS's webpack config
• Fix pending TODOs DecryptionParams may be passed when using queryHistory instead of just keys. #927
• Remove console.log files refactor: remove console.log usage #926
• Make Waku protocol optional in Waku (involves defining and creating interfaces). Allow disabling of Relay #784
• Set interface for Waku protocols
• Make js-waku mono repo
  • extract peer dns discovery in separate package feat!: extract utils, dns discovery and enr from core #995
  • extract create Waku (default settings) in separate package
  • extract Waku message version 1 in separate package (involve defining interfaces) feat!: extract version 1 encryption to @waku/message-encryption #999
• Release
• Updates examples core: bump @waku/core to 0.0.6 examples.waku.org#147

[fryorcraken]

Work towards first release: #891

[fryorcraken]

Note regarding the usage of @waku/core instead of js-waku.

For now, I am exporting the different modules via the exports map.

Once we have a release of js-waku that works and makes sense (meaning that all functionality are re-exposed and usable from a consumer pov) then we can more the repo to multi-repo, extract the various exports paths to separate packages and create @waku/core at this point with an array of other packages.

[fryorcraken]

Migration to monorepo has started:

• chore: migrate to monorepo #987
• chore: setup auto-release #988
• chore: setup semantic-release for workspace #989

[fryorcraken]

@waku/create and @waku/interfaces split away: #990

[fryorcraken]

• feat!: extract utils, dns discovery and enr from core #995
• feat!: extract version 1 encryption to @waku/message-encryption #999

[fryorcraken]

Next steps:

• API Improvements #1010
• Update docs