feat: registryCustomizations (#1217)

* chore: tsconfig for top-level await

* feat: registry variants

* feat: merge custom registry in registryAccess

* feat: customRegistry by default via registryAccess

* feat: decompose using any uniqueIdElement

* chore: remove license

* test: variants nut test fixture for permset

* chore: replacement for baseName that handles dots in fullName

* refactor: handle dots for fieldPermissions

* chore: log presence of customizations

* refactor: extract polling options logic

* test: snapshots for custom object conversion

* test: move snapshot location

* test: snapshots for staticResource

* test: snapshots for both sample repos

* test: sample changed?

* chore: vscode setting

* test: better test labeling

* test: non-dynamic mocha tests

* test: clear snaps

* test: improve file naming

* test: re-commit snapshots

* chore: no linter noise inside snapshots

* style: preserve snapshot data

* style: don't change snapshots

* chore: prettier ignore snapshots

* chore: didn't use prettier

* test: labels project, put sfdx-project where they go

* test: snapshot for partial permSet decomposition

* test: moved to snapshots

* test: include force-app or other top-level project folder

* feat: option to specify non-cwd projDir for CSBuilder

* fix: pass registry in more places

* test: ability to compare xml files by parsing to json

* test: permissionset decomposition works

* feat: support for startEmpty/topLevel with decomposed

* test: use absolute paths to fix labels snapshot

* test: changes for or caught by UT

* test: territory2 snapshots from pdr

* test: snapshot for deb

* test: snapshots for folder types

* test: snapshot testing docs and dir comparison utility

* test: snapsnots in CI only on linux.  Run locally

* test: snapshots still run as part of `yarn test`

* ci: qa for workflows change

* ci: longpaths in more places

* ci: longpaths for windows perf test

* test: longpaths only for windows

* chore: script change

* test: identical asserts

* test: snapshots for workflows

* test: workflow variant

* feat: prefer xmlElement over directory when present

* test: snapshot for decomposed workflow via variant

* test: p.xml in the correct location for workflow mdapi

* feat: presets with snapshot for PS preset

* fix: handling no presets

* refactor: empty defaults for customization/presets

* fix: pass empty array for customizations

* test: decomposed types are mostly not addressable to keep out of p.xml

* feat: preset for decomposed permsets

* test: snapshots for using permset preset

* docs: notes about registry variants

* test: sharingRules

* fix: correct parent for fieldPermissions

* chore: tsconfig for presets json

* refactor: emit empty parents

* feat: sharingRules presets

* test: snapshot for decomposedSharingRules

* test: keep empty workflow

* test: use main workflow branch

* test: snapshot for bots

* ci: longpath PR was merged, use main

* chore: type-only import for jsforce

* chore: bump core for xnuts

* ci: test uses main

* test: refactor metadata transfer and tests using function

* test: generated xml order is determinstic and may not match original

* chore: bump core

* test: snapshot fix for workflow variants

* test: account workflow snapshot update

* test: workflow via presets

* test: snapshot for empty top-level file

* test: decomp for labels

* refactor: restore dot-joined files (ex: FieldPermissions)

* test: snapshot of top-level-only labels

* test: ut changes for empty file behavior

* test: adjust snaps for xml ordering of recompose

* docs: todo updates

* ci: test only exclude snapshot tests

* test: dir names as const

* docs: docs for 3 presets

* test: remove toplevel  workflow from source snapshot

* test: apply registryValidation tests to presets

* test: dynamically load the preset folder

* refactor: import what you need

* refactor: import cleanup

* test: remove original variant nut in favor of snapshot testing

* test: full diff for windows debug

* docs: comments

* feat: deterministic md recomposition order

* feat: deterministic md recomposition order

* chore: sort order for recompose

* test: deep equal in any order

* test: order of __c vs letters

* test: underscores first

* test: more underscore first

* chore: docs and export cleanup

* refactor: avoid unnecessary compSet

* chore: console.log cleanup

* test: export dirents

* test: mpd project for snapshots (3 dirs)

* test: snapshot for recomposed mdapi

* test: identical dirs must exist

* chore: snapshots for 3 original dirs after applying xml over them

* test: force-app needs to go back into its original dir

* style: use empty element, not self-closing tag

* test: copy to a tmpDir

* chore: compare relative paths for identical dirs

* chore: undo snapshots for source merging

* chore: cleanup for test directories

* test: snapshot for the force-app dir

* test: snapshot for dir 2

* test: snapshot for foo-bar/app

* chore: comments for unused logger fn

* docs: code comments instead of pr comments

* test: settings suffix list

* refactor: explicit return for void

* refactor: avoids void 0

* refactor: types match fs

* refactor: follow complexity rules

* chore: refactor test callCount

* fix: componentSet.map isn't like array.map

* refactor: no tooling api for diagnostic utils

* refactor: thinner methods

* refactor: metadata resolver functions

* fix: don't save empty parent objects

* refactor: fewer logic branches

* refactor: remove 2 unused props from type

* refactor: use true as only foundMerge option

* refactor: extract writeInfo logic for children, parents

* refactor: remove dangerous assertion

* test: top-level types cannot also be children

* style: jsdoc indent for windows ut

* chore: cleanup old scripts and docs

* fix: adapter names

* refactor: more coe cleanup

* test: ut for nondecomposed

* fix: handling for wildcards matching org contents

* refactor: build metadata map only when needed

* refactor: private methods to functions

* refactor: simplify partialDelete handling

* fix: handles colons in metadata names

* chore: bump core

* refactor: more registries to component sets, eliminate an unneeded one

* chore: name typo

* test: change test to spy on a function

* refactor: put unaddressable children in results

* refactor: spread not concat

* refactor: cleaner cs and csb

* chore: wireit watches preset changes

* feat: handles "partial delete" of decomposed children

* refactor: reorganize retrieve code

* style: formatting

* refactor: remove barrel

* refactor: deployResult logic

* perf: dedupe deploy results once

* chore: comments about removing enums

* chore: script/dep cleanup

* chore: remove logFn

* chore: cleanup from pr review

* fix: getZipBuffer's converter needs custom registry

* fix: decomposed children that aren't addressable cause their entire parent to deploy

* chore: remove logFn

* chore: add decompsed CustomLabels

* chore: bump kit

* chore: labels preset follows registry rules

* refactor: change preset names to beta

* fix: confirm DCL behavior, fix snapshots, add .md entry

* docs: fix extension

* chore: add comment explaining new error [skip-ci]

---------

Co-authored-by: Willie Ruemmele <[email protected]>

.sfdevrc.json

@@ -3,14 +3,14 @@
     "compile": {
       "clean": "if-file-deleted",
       "command": "tspc -p . --pretty --incremental",
-      "files": ["src/**/*.ts", "src/registry/*.json", "**/tsconfig.json", "messages/**"],
+      "files": ["src/**/*.ts", "src/registry/**/*.json", "**/tsconfig.json", "messages/**"],
       "output": ["lib/**", "*.tsbuildinfo"]
     },
     "test": {
       "dependencies": ["test:only", "test:compile", "test:registry-validation", "test:snapshot"]
     },
     "test:only": {
-      "command": "nyc mocha \"test/**/*.test.ts\" --exclude \"test/registry/registryValidation.test.ts\" --exclude \"test/snapshot\"",
+      "command": "nyc mocha \"test/**/*.test.ts\" --exclude \"test/registry/registryValidation.test.ts\" --exclude \"test/snapshot/**\"",
       "env": {
         "FORCE_COLOR": "2"
       },

HANDBOOK.md

@@ -102,8 +102,6 @@ The config file consists of a handful of different indexes.
 
 This file is large and luckily, not entirely crafted by hand. And because new metadata types are being added to the platform each release, we’ll need to update the [Metadata registry file]. The update-registry module in the scripts folder automatically updates the registry as best it can using a describeMetadata() call against a provided Salesforce org, without overwriting manual changes. It also attempts to update the indexes listed in the previous section. When generating a new version of the registry, it’s important to manually review the changes to ensure they make sense and aren’t destructive. When in doubt, test functionality with the new version. See [Contributing Metadata Types to the Registry](./contributing/metadata.md) in the development README on how to invoke the script with Yarn.
 
-Unfortunately. we sometimes need to manually change a type definition, albeit rarely. The `typeOverride.json` file allows us to overwrite any updates the script attempts to make that we don’t want to happen.
-
 🛠 _Another issue is we are limited by the permissions and licenses of the org that we are running the update script on, which may return incomplete describe information. We need to address this as soon as possible to not run into type gaps between releases. This is being worked on as of 7/16/2021._
 
 ### Breaking Down a Metadata Type Entry
@@ -148,6 +146,32 @@ registry.types.auradefinitionbundle.directoryName; // => 'aura'
 
 📝 _The registry object is “deeply frozen”, meaning none of its properties, even the nested ones, are mutable. This is to ensure that a consumer cannot change registry information in a process and potentially affect functionality._
 
+### Registry Variants
+
+It's possible that the registry isn't what a user wants
+
+- you're building a new type and want an easy way to test your registry changes
+- you disagree with the choice made in the registry (ex: I wish PermissionSet was decomposed instead of a giant file)
+
+There are two options available if you're in an sfdx project.
+
+- `registryCustomizations`: add your own partial of the the registry that'll be merged into the registry. It's a json object just like the registry itself.
+- `registryPresets`: SDR defines "preset" registryCustomizations and you say which ones you want applied. Each one is a partial, and you can have any or all of them by listing them. Names correspond to something in src/registry/presets, so your project file could use
+
+```json
+  "registryPresets": ["decomposePermissionSetBeta", "decomposeSharingRulesBeta"]
+```
+
+if you want only those 2 presets.
+
+The naming convention is `decomposeFoo` where `Foo` is the top-level metadata type, and refers to `/presets/Foo.json`.
+
+This requires that any constructedRegistry know about your project directory. You'll see lots of code passing that around, and passing around constructed RegistryAccess to avoid getting the default if a custom one was already constructed.
+
+Be careful when instantiating classes (ex: ComponentSet) that will default a Registry or RegistryAccess--make sure to pass in a registry to preserve customizations.
+
+**Updating presets** If you do need to update a preset to make a breaking change, it's better to copy it to a new preset and give it a unique name (ex: `decomposeFooV2`). This preserves the existing behavior for existing projects with the old preset.
+
 ### Querying registry data
 
 While it’s perfectly fine to reference the registry export directly, the `RegistryAccess` class was created to make accessing the object a bit more streamlined. Querying types and searching the registry is oftentimes easier and cleaner this way and contains built-in checking for whether or not a metadata type exists. Here’s a comparison of using each:
@@ -683,7 +707,7 @@ Oftentimes we need to work with a unique collection of components. A `ComponentS
 
 Component sets key components using their fullName and type id, i.e. only one pair will ever be present in the set. This implies that anything that conforms to the `MetadataComponent` interface can be added to a set. `SourceComponent`s in fact have particular logic on how they are stored. While only one fullName and type pair can be present in a set, multiple source-backed components can map to the same pair under-the-hood. This is how splitting a CustomObject across multiple package directories is achieved - by treating each version of the component as a separate source-backed component that points to the same fullName and type pair in a set. When component merging happens, the conversion logic is able to combine the source components belonging to the same pair.
 
-The easiest way to build `ComponentSets` is by using the `ComponentSetBuilder` class's static `build()` function.  See <https://github.com/forcedotcom/source-deploy-retrieve/tree/main/src/collections/componentSetBuilder.ts> for details.  Simply pass a configuration object to `build()` and it takes care of the rest, including handling destructive changes!
+The easiest way to build `ComponentSets` is by using the `ComponentSetBuilder` class's static `build()` function. See <https://github.com/forcedotcom/source-deploy-retrieve/tree/main/src/collections/componentSetBuilder.ts> for details. Simply pass a configuration object to `build()` and it takes care of the rest, including handling destructive changes!
 
 Let’s look at some examples of adding components and testing membership:
 
@@ -827,7 +851,7 @@ The order in which we’ve examined the concepts up to this point has been inten
 
 Metadata API deploys and retrieves are asynchronous operations. Once a request has been made, the status of the operation needs to be polled to determine whether or not the request has finished processing. This lifecycle is encapsulated with the `MetadataApiDeploy` and `MetadataApiRetrieve` classes — one instance is meant to map to one operation. They expose useful methods to process the lifecycle and make other requests, which will be illustrated later. These both extend `MetadataTransfer` in order to share common functionality such as polling.
 
-The most common and simplest way to deploy and retrieve metadata is by using a [Component Set](#component-sets).  The easiest way to build `ComponentSets` is by using the `ComponentSetBuilder` class's static `build()` function.  See <https://github.com/forcedotcom/source-deploy-retrieve/tree/main/src/collections/componentSetBuilder.ts> for details.
+The most common and simplest way to deploy and retrieve metadata is by using a [Component Set](#component-sets). The easiest way to build `ComponentSets` is by using the `ComponentSetBuilder` class's static `build()` function. See <https://github.com/forcedotcom/source-deploy-retrieve/tree/main/src/collections/componentSetBuilder.ts> for details.
 
 ### Establishing an org connection
 

METADATA_SUPPORT.md

@@ -8,10 +8,8 @@ Currently, there are 0/0 supported metadata types.
 For status on any existing gaps, please search or file an issue in the [Salesforce CLI issues only repo](https://github.com/forcedotcom/cli/issues).
 To contribute a new metadata type, please see the [Contributing Metadata Types to the Registry](./contributing/metadata.md)
 
-|Metadata Type|Support|Notes|
-|:---|:---|:---|
-
-
+| Metadata Type | Support | Notes |
+| :------------ | :------ | :---- |
 
 ## Next Release (v61)
 
@@ -20,7 +18,7 @@ To contribute a new metadata type, please see the [Contributing Metadata Types t
 
 ## Additional Types
 
-> The following types are supported by this library but not in the coverage reports for either version.  These are typically
+> The following types are supported by this library but not in the coverage reports for either version. These are typically
 >
 > 1. types that have been removed from the metadata API but were supported in previous versions
 > 1. types that are available for pilots but not officially part of the metadata API (use with caution)

Registry Variants TODO.md

@@ -0,0 +1,29 @@
+there's a bug with converter.convert when converting mdapi to source with a long output directory--it'll use the entire directory for customLabels names (see snapshots inside the customLabels test)
+
+Other types seem to get a relative-to-proj path.
+
+Probably doesn't affect the CLI since we normally run the converts from inside a project rather than providing a long proj path.
+
+---
+
+## Open bugs
+
+## QA
+
+1. partial delete: deleting a member locally redeploys without that file
+1. partial delete: "removing" a member remotely deletes it locally if present
+1. conflicts get assigned to the correct place (both directions)
+
+---
+
+## doc work
+
+1. Contributing Metadata (how to test new types using variants before adding to the real registry)
+1. Migration process (convert, delete, convert?)
+
+---
+
+## Features
+
+1. how does it work with packaging
+1. how does it work with STL

package.json

@@ -26,7 +26,7 @@
   },
   "dependencies": {
     "@salesforce/core": "^6.7.3",
-    "@salesforce/kit": "^3.0.15",
+    "@salesforce/kit": "^3.1.0",
     "@salesforce/ts-types": "^2.0.9",
     "fast-levenshtein": "^3.0.0",
     "fast-xml-parser": "^4.3.5",
@@ -47,15 +47,12 @@
     "@types/graceful-fs": "^4.1.9",
     "@types/mime": "2.0.3",
     "@types/minimatch": "^5.1.2",
-    "@types/proxy-from-env": "^1.0.4",
-    "@types/shelljs": "^0.8.15",
     "deep-equal-in-any-order": "^1.1.19",
     "deepmerge": "^4.3.1",
     "eslint-plugin-sf-plugin": "^1.17.4",
     "jsforce": "^2.0.0-beta.29",
     "mocha-junit-reporter": "^1.23.3",
     "mocha-snap": "^5.0.0",
-    "shelljs": "0.8.5",
     "ts-node": "^10.9.2",
     "ts-patch": "^3.1.1",
     "typescript": "^5.4.2"
@@ -78,7 +75,7 @@
     "prepare": "sf-install",
     "repl": "node --inspect ./scripts/repl.js",
     "test": "wireit",
-    "test:nuts": "mocha \"test/nuts/local/**/*.nut.ts\" --timeout 500000",
+    "test:nuts": "mocha \"test/nuts/local/**/*.nut.ts\" --timeout 500000 --parallel --job 20",
     "test:nuts:suggest": "mocha \"test/nuts/suggestType/suggestType.nut.ts\" --timeout 10000",
     "test:only": "wireit",
     "test:registry": "mocha ./test/registry/registryCompleteness.test.ts --timeout 50000",
@@ -108,7 +105,7 @@
       "command": "tspc -p . --pretty --incremental",
       "files": [
         "src/**/*.ts",
-        "src/registry/*.json",
+        "src/registry/**/*.json",
         "**/tsconfig.json",
         "messages/**"
       ],
@@ -165,7 +162,7 @@
       "output": []
     },
     "test:only": {
-      "command": "nyc mocha \"test/**/*.test.ts\" --exclude \"test/registry/registryValidation.test.ts\" --exclude \"test/snapshot\"",
+      "command": "nyc mocha \"test/**/*.test.ts\" --exclude \"test/registry/registryValidation.test.ts\" --exclude \"test/snapshot/**\"",
       "env": {
         "FORCE_COLOR": "2"
       },