[heft] Add Heft "Phases" and "Tasks" to Allow for Heavy Customization of Heft Build Flows

[D4N14L]

Summary

Implements the design discussed in #3181. Rejigs how plugins are developed for Heft. Rejigs how plugins interact with the "build" flow.

Details

As stated in the summary, many of the changes here are implementations of the designs discussed in #3181. A few of the bigger callouts for this change include:

• heft.json changes
  • Replaced the existing schema with a new format that allows for developers to customize Heft behaviour and to create/modify Heft actions (ex. heft build). These actions are made up of phases and tasks.
    • Phases contain tasks, and can take dependencies on other phases
      • Phases become the name for Heft actions (ex. heft build will run all phases up to and including the build phase, as defined in heft.json)
      • Phases are executed in dependency order
      • Cleaning for all tasks in a phase occurs at the beginning of each phase execution when invoked with the --clean parameter
    • Tasks contain either a single Heft plugin or a single Heft event and can take dependencies on other tasks within the same phase.
      • Tasks are executed in dependency order
• heft run action
  • Allows for scoped executions of Heft phases using Rush-like scoping arguments
  • heft run --to <phaseName> and heft run --only <phaseName> scoping parameters can be used to target a limited number of phases
  • Passing parameters to the phases done as heft run --to phaseName -- --parameter
• heft <phaseName> actions
  • Equivalent to running heft run --to <phaseName> --
• Heft plugin development
  • Now two types of plugins: lifecycle plugins and task plugins
    • Task plugins can only be used with tasks, and provide a clean and a run hook
    • Lifecycle plugins which can only be specified in heft.json heftPlugins field, and provide a clean, toolStart, toolStop, and metrics hook
  • Plugin packages must now specify a heft-plugin.json manifest in the root of the package
    • Points to all plugin entrypoints and allows for specification of multiple plugins within a package
    • Allows for specification of a schema file to validate heft.json-provided options
    • Allows for specification of CLI parameters
      • Each parameter is registered under two different names: --parameter and --PluginName:parameter
        • When multiple plugins provide the same parameter, the former parameter is excluded from the CLI
        • The latter parameter should always be available on the CLI
• Miscellaneous changes
  • Unique temporary and cache folders are now provided to plugins via the HeftTaskSession object
    • temp/lifecycle.<lifecycleplugin> and temp/<phasename>.<taskname> folders are deleted when cleaning
    • .cache/lifecycle.<lifecycleplugin> and .cache/<phasename>.<taskname> folders are deleted when cleaning and --clean-cache is specified
  • clean hooks can provide files to delete via the options passed into the hook
  • run hooks can provide files to be copied via the options passed into the hook
  • Exposes the IRigToolResolver API to obtain packages from rigs the same way TypeScript or API Extractor are loaded
  • Move testing-related configuration options to @rushstack/heft-jest-plugin

Changes made that are used by (but external to) Heft:

• Allow loading an API Extractor configuration without resolving and validating the existence of target files
  • Used to load up the API Extractor configuration before target files exist, which is done for the APIExtractorPlugin clean hook
• Allow for adding multiple documented synonyms for ts-command-line parameters
  • Used for plugin parameter conflict resolution
• Add FileError to @rushstack/node-core-library
• Add SubprocessTerminator to @rushstack/node-core-library
  • Other Subprocess-related classes were removed from Heft, as their implementation was tied closely to Heft and is no longer being used

TODO:

• Improve usage of metrics lifecycle hook
• Re-implement heft start functionality
  • Internal design discussion ongoing, will update [heft] Design Proposal: Alignment with Rush phased commands #3181 with design thoughts
• Add documentation

How it was tested

Updated most of repository to build using new Heft. Remaining unconverted projects use Heft as a cyclic dependency and will be converted after the new version of Heft is published. Updated build tests to target new Heft.