Replace npm with yarn

[andrew-codes]

• Closes Proposal: Replace npm with yarn #5477

User facing changelog

N/A

Additional details

Things affected by this change:

• all npm scripts; i.e. yarn test-unit, etc.
  • all scripts are must run from root, but do provide a way to run within the context of one or more packages
  • see common top level tasks for more details
• deps explicitly added to packages depending on them; mocha, eslint, and possibly a few others
  • the process of adding mocha was not straight-forward as there exist several different versions in the code-base. Tried to leverage the lowest common denominator to reduce risk
• removed several ./scripts files that became unnecessary

Details

With yarn workspaces optimizing where node_modules live, we can no longer rely on a node_module existing in a particular node_module directory (root or package level). There were a few places that manually reached into node_modules to read a file, etc. that required updates. One such way to update was to introduce the resolve-pkg which a) allows for packages to be resolved within the context of a directory and b) does not require the main field in the resolved packages' package.json

I also updated some root scripts to enable running commands from the root more quickly. New packages should be added in the root and not in the package sub-dir.

How has the user experience changed?

No user experience changes, but developers of cypress will be required to use yarn instead of npm.

PR Tasks

• Measure performance impact
• Update renovate.json
• Update READMEs and other forms of docs

Developer Experience

I think there are two primary use cases to watch out in terms of developer experience; validation and debugging.

Validation means I want to validate my changes, i.e. I want to run tests in a package or packages to verify that my changes work. Validation tasks intentionally execute at critical points in development. It is not necessary to run validation tasks with every change a developer makes.

The second is debugging; running tests and stopping the process for inspection. In this use case, it would be ideal for the debugging process to run at the beginning of my development cycle, with it re-running relevant tests as I change files. However, running tests in watch mode is not sufficient, as in this use case, I also want to stop the running process (breakpoint) to inspect/debug code line by line.

• debugging experience via terminal; limited to console.logs by running all tests of a specific package; if the below doesn't pan out, then I will investigate running individual files via CLI
• debugging experience via IDE/editor; deferring this to investigate wallabyjs and other possible solutions

Using Yarn

• update root package engines to include yarn
• replace npm run with yarn in package.json scripts
• remove -- --arg; yarn will pass args without the need for --

Using Yarn Workspaces

Maintain check-deps functionality. Use case is: a developer changes branches and forgets to re-run yarn. Check-deps will error out and alert the developer that yarn is necessary before running other scripts.

Note: building optimizations (only build packages that have changed) are deferred in this PR.

• remove direct access to node_modules directory contents
  • post-install script
  • importing of SCSS from node_modules
• remove unneeded check-deps
• maintain check-deps functionality (we are not optimizing building; only installing of packages)
  • this can be accomplished via yarn check --integrity; see yarn docs for more details
  • if the desire is to warn the user of missing deps when switching branches, etc.) then we could simply automatically install packages if yarn check --integrity ever fails.

Using Lerna

• replace root package scripts with those that can run across packages
  • test-unit
  • test-integration
  • test-e2e
  • watch
  • prebuild
  • build
  • clean-deps
  • lint-coffee
  • remove all npm task and associated scripts
    • replace npm run all in root package.json
    • remove run.js and related scripts
    • update circle.yml
    • update appveyor.yml
• replace root package scripts to not reach into packages to run commands
• Remove bin-up packages and references
  • cli
    • linting running npm run lint already has errors before the introduction of yarn

---

Measuring Performance

npm

Use Case	Commands	Measurement
without prior node_modules	find . -name node_modules -type d -prune -exec rm -rf {} \; && time npm i	4m 43.167s
previously installed node_modules	find . -name node_modules -type d -prune -exec rm -rf {} \; && npm i && time npm i	3m 29.706s
installing a new node_module; lerna	find . -name node_modules -type d -prune -exec rm -rf {} \; && npm i && time npm i react --dev	0m 31.826s
building	time npm run build	1m 20.562s
testing	time npm run all test-unit	0m 40.485s
size on disk	du -ch .	2.3GB

yarn

Use Case	Commands	Measurement
without prior node_modules	find . -name node_modules -type d -prune -exec rm -rf {} \; && time yarn	1m 44.496s
previously installed node_modules	find . -name node_modules -type d -prune -exec rm -rf {} \; && yarn && time yarn	1m 28.837s
installing a new node_module; lerna	find . -name node_modules -type d -prune -exec rm -rf {} \; && yarn && time yarn add -W --dev react	0m 52.830s
building	time yarn build	1m 05.780s
testing	time yarn test-unit	0m 35.109s
size on disk	du -ch .	1.7GB

Notes

Removing direct access to node_modules

Node_modules is an opaque structure. We should not reach directly into it to grab files. This issue is highlighted with the introduction of yarn workspaces. Any given node_module may or may not live in its dependent package's node_modules directory. This is because yarn will optimize node_modules; potentially hoisting them into the root of the mono-repo.

An example of this issue:

# adding workspaces and running yarn to install node_modules
yarn

# produces
error /Users/andrewsmith/developer/repos/cypress-yarn/node_modules/cypress: Command failed.
Exit code: 1
Command: node ./scripts/post-install.js
Arguments:
Directory: /Users/andrewsmith/developer/repos/cypress-yarn/node_modules/cypress
Output:
set -e
cp -R node_modules/@types/blob-util types
/Users/andrewsmith/developer/repos/cypress-yarn/node_modules/shelljs/src/common.js:401
      if (config.fatal) throw e;
                        ^

Error: cp: no such file or directory: node_modules/@types/blob-util
    at Object.error (/Users/andrewsmith/developer/repos/cypress-yarn/node_modules/shelljs/src/common.js:110:27)
    at /Users/andrewsmith/developer/repos/cypress-yarn/node_modules/shelljs/src/cp.js:246:14
    at Array.forEach (<anonymous>)
    at Object._cp (/Users/andrewsmith/developer/repos/cypress-yarn/node_modules/shelljs/src/cp.js:243:11)
    at Object.cp (/Users/andrewsmith/developer/repos/cypress-yarn/node_modules/shelljs/src/common.js:384:25)
    at /Users/andrewsmith/developer/repos/cypress-yarn/cli/scripts/post-install.js:20:9
    at Array.forEach (<anonymous>)

This is because the postinstall script attempts to read files within the @types/blob-util package. This package has been optimized, hoisted to the root of the repo, and no longer guaranteed to be located in either the package or the root.

[cypress-bot]

Thanks for the contribution! Below are some guidelines Cypress uses when doing PR reviews.

• Please write [WIP] in the title of your Pull Request if your PR is not ready for review - someone will review your PR as soon as the [WIP] is removed.
• Please familiarize yourself with the PR Review Checklist and feel free to make updates on your PR based on these guidelines.

PR Review Checklist

If any of the following requirements can't be met, leave a comment in the review selecting 'Request changes', otherwise 'Approve'.

User Experience

• The feature/bugfix is self-documenting from within the product.
• The change provides the end user with a way to fix their problem (no dead ends).

Functionality

• The code works and performs its intended function with the correct logic.
• Performance has been factored in (for example, the code cleans up after itself to not cause memory leaks).
• The code guards against edge cases and invalid input and has tests to cover it.

Maintainability

• The code is readable (too many nested 'if's are a bad sign).
• Names used for variables, methods, etc, clearly describe their function.
• The code is easy to understood and there are relevant comments explaining.
• New algorithms are documented in the code with link(s) to external docs (flowcharts, w3c, chrome, firefox).
• There are comments containing link(s) to the addressed issue (in tests and code).

Quality

• The change does not reimplement code.
• There's not a module from the ecosystem that should be used instead.
• There is no redundant or duplicate code.
• There are no irrelevant comments left in the code.
• Tests are testing the code’s intended functionality in the best way possible.

Internal

• The original issue has been tagged with a release in ZenHub.

[flotwig]

• verified macOS build (Gleb)
• verified windows build runs
  • verified 32-bit build is 32-bit:
    
  • verified 64-bit build is 64-bit:
    
• verified linux build

LGTM, nice work @andrew-codes