Convert to monorepo

[mcmire]

Remaining work

• Look into why some tests involving bn.js aren't working (see https://github.com/MetaMask/controllers/pull/831/files#r898282054) not necessary

Completed work

• Create monorepo file structure by moving all modules to packages/
• Instruct Yarn to treat packages/* as workspaces
• Add Yarn constraints for root-level and package-level package.jsons
• Make use of TypeScript's project references to connect all packages together into one build
• Move Jest config to each package and ensure that all tests pass
• Ensure that ESLint still works (with minimal changes)
• Move TypeDoc config to each package and ensure that it still works
• Update Yarn constraints to synchronize all dependencies and prevent duplicate dependencies
• Instead of symlinking scripts (e.g. validate-changelog, test) use paths (e.g. ../../scripts/*)
• Update each package's README to add a basic description, remove usage, and update Contributing section to link to root README's Contributing section
• Bring Release & Publishing section in root README up to date with current monorepo release process
• Update PreferencesController to use a separate instance of ContactEntry instead of sharing ContactEntry from AddressBookController
• Split AddressBookController from PreferencesController into separate packages
• Split PhishingController and EnsController into separate packages
• Go back through packages/controller-utils/src/util.ts and extract any functions to their respective packages
• Use Jest config to use customExportConditions instead of transformIgnorePatterns to use CJS exports from dependencies instead of ESM (see message in Slack). Shouldn't need to use allowJs or ts-jest/presets/js-with-ts either.
• Update description in each package's package.json to match the README
• Check every file and ensure that contents are listed in the same order that they appeared in the original source file
• Look back through the old version of index.ts and make sure everything is being exported that should be, and nothing is being exported that shouldn't
• Add Yarn constraints to verify that name of each package matches its directory and that its homepage matches the name
• Address Bug when creating a branch for a non-initial release for a monorepo with independent versions create-release-branch#43
• Fix publish-release step to take advantage of caching (Use cached dependencies and immutable install during publish #942)

Deferred to a future PR

• Rename "collectible" references to NFTs (e.g. CollectiblesController, NftsController) — done in Rename "Collectible" to "NFT" #934
• Add a reference to root README to high-level things like how to make a controller, etc.
• Add Usage for all controllers
• Investigate whether we can Yarn workspaces for compiling TypeScript instead of TS project references so that we don't have to have so many tsconfig files
  • Use yarn workspaces foreach --topological-dev exec "echo '' >/dev/null" to check for circular dependencies first
• Look into how Yarn produces documentation and see if we can replicate
• Remove EnsController since it doesn't seem to have been exported
• Fix NetworkController tests so we don't have to run --forceExit
• Cache *.tsbuildinfo files in CI so that yarn build doesn't take as long for subsequent runs
• Use cross-fetch instead of isomorphic-fetch
• Remove hexToText
• Test on Node 18.x and 19.x like the module template
• Use Json from @metamask/utils
• Use fromHex and toHex (or their equivalents) from @metamask/utils
• Prefer @ts-expect-error over any

How to review this PR

In addition to the comments I've left, I would check this branch out locally, and then:

• Run yarn build. This should compile the TypeScript files in each package and output them to dist/ in the package's directory.
• Run yarn test. This should run all tests for all packages.
  • Constraints are specified in constraints.pro. This allows us to make sure that package.json at the root and also at the per-package level have the right fields with the right values.
• Take a look at how the TypeScript config is structured:
  • When writing TypeScript code, your IDE will use tsconfig.json. This makes use of project references to include all TypeScript code across all packages in a way that informs TypeScript which packages depend on which other packages. Per-package configuration is kept at packages/*/tsconfig.json, and all of these files inherit from tsconfig.packages.json.
  • When running yarn build, tsconfig.build.json will be used instead. This also uses project references, but brings in packages/*/tsconfig.build.json for each package instead. All of these files inherit from tsconfig.packages.build.json and exclude test-specific files from the build. TypeScript will consult each package's TypeScript config to figure out where to output files (outDir is relative to the package, not the root).
• Take a look at how Jest is configured:
  • Each package has its own jest.config.js (previously jest.config.ts), with common options extracted to jest.config.packages.js in the root. The tests for each package are run individually. All tests are run using yarn workspaces foreach to iterate through all packages.
  • We have configured ts-jest to run not only TypeScript files but also JavaScript files through the TypeScript compiler (see the preset setting in jest.config.packages.js). This is necessary because there are some dependencies (multiformats, ethjs-*) which ship with uncompiled, and sometimes Babel-specific, code. I am not sure how we got around this before, but these files need to be transformed in order to be used at all. Update: It's true that we have dependencies that ship uncompiled but we've gotten around this by using customExportConditions in the Jest config file.
  • The above constraint means that we need a special tsconfig.test.json for tests, which sets allowJs to true. This means there are actually three levels of tsconfigs: dev, build, and test. Update: This is no longer the case

[mcmire]

Pushed up some more fixes. Outstanding issues remaining:

• The coverage report isn't being generated anymore. Seems that Jest projects isn't compatible with the coverage settings: Tests from different monorepo package are ignored in coverage jestjs/jest#9323
• Still need to figure out TypeDoc.
• Need to bring changelogs over.

[mcmire]

• Tests now work, although there are some that fail intermittently due to timers still being active. I've temporarily addressed this by running yarn jest with --forceExit, but we'll probably want to fix this later (by introducing a withController helper in tests that construct controllers that need to be destroyed later).
• Also TypeDoc works.
• Still need to move changelogs.

[Gudahtt]

LGTM! A couple of small things left to improve but nothing major

[Mrtenz]

Great work!