Unexpected SFDX plugin hooks interface changes

[gdman]

Summary

Does anyone know anything about changes to the SFDX plugin hooks? It appears the latest version of SFDX includes the new source plugin that fundamentally changes the interface for the predeploy hook. Example of the new format below.

Does anyone know of anywhere where I can find out more about these changes, an announcement or such like? It looks like the documentation hasn't been updated - https://developer.salesforce.com/docs/atlas.en-us.sfdx_cli_plugins.meta/sfdx_cli_plugins/cli_plugins_customize_hooks_list.htm.

SourceComponent {
    name: 'Account',
    type: {
        id: 'flow',
        name: 'Flow',
        suffix: 'flow',
        directoryName: 'flows',
        inFolder: false,
        strictDirectoryName: false
    },
    xml: '/Users/repository/force-app/main/default/flows/Account.flow-meta.xml',
    parent: undefined,
    content: undefined,
    _tree: NodeFSTreeContainer {},
    forceIgnore: ForceIgnore { DEFAULT_IGNORE: [Array] }
}

This is a breaking change for one of my plugins, if anyone else has plugins on predeploy (or I'd assume other hooks too) - heads up they may also be broken.

[mshanemc]

Our pinned announcements are helpful to get advance warnings.
#1057

The release notes are a good way to stay informed of what's closer to shipping https://github.com/forcedotcom/cli/blob/main/releasenotes/README.md#71120-august-4-2021

I think we need to change the docs around the source hooks @jshackell-sfdc
https://developer.salesforce.com/docs/atlas.en-us.sfdx_cli_plugins.meta/sfdx_cli_plugins/cli_plugins_customize_hooks_list.htm

[gdman]

@mshanemc thanks for the info.

Is there a way of changing the contents of the package before it gets zipped up and sent to the MD API with the new hooks/source plugin? I ask because that's what I was doing before with my https://github.com/gdman/sfdx-destruction plugin i.e. copying in destructiveChangesPre and Post files.

Alternatively, are there plans for native support of destructive changes in the source plugin? There is an issue that tracks it here: #188 but it's not been updated in a while.

[mshanemc]

We've starting that destructive changes work. You'll see it in the library first, then in the source plugin.

I'm gonna defer to @shetzel for how the new hook would would work with the new deploy process.

[shetzel]

@gdman - sorry for my delayed response and the breaking hook changes. About the hook changes, we had to make a decision on whether to keep backwards compat with hook payloads and negate many of the performance increases that come from using the source-deploy-retrieve (SDR) library, or break compat and change the hook payloads. We never want to break backwards compatibility but in this case we thought the pros outweighed the cons. With the SDR library there isn't any intermediary file system writing during the deploy anymore, so copying in destructive changes aren't possible. However, I recently added support to the SDR library for destructive changes to ComponentSets. If the hook were to add the ComponentSet as part of the payload you could modify that and it should get you the same behavior.

As for your other question, native support of destructive changes is definitely coming soon. I'd say it's coming this Fall or Winter time frame. Adding the support to the SDR library was the first step. Out of curiosity, what would solve your usecases for natively supporting destructive changes? Would this be as simple as adding a new flag for destructive changes file path to source:deploy command? We've also been kicking around the idea of trying to maintain a destructiveChanges list per project as the CLI encounters them so they could be included in specific deploys, such as to UAT sandboxes and production. Otherwise you're doing source control branch comparisons when maintaining your various environments (development, UAT, staging, production) to keep track of destructive changes.

[gdman]

@shetzel - Completely understand the decision to change the hook payloads, it makes total sense.

The issue was more that it was done without any deprecation or warning. As a plugin developer, the first I time I found out about it was when the new version had already been released and I was receiving messages to say the plugin didn't work. The only way I could have known about it sooner would have been to be checking the release notes for SFDX regularly.

In future, maybe put out an update with something that puts a message out to stdout/stderr saying that functionality is going to stop working in the next release. Maybe better still would be to have a new version of the hook, put a message on the old one to say it's no longer supported and direct people to the new one. Just changing the payload could have led to something worse than an error from trying to split a null string.

This feels like a one time kind of event so probably not something to worry too much about but knowing interfaces could change without warning is the sort of thing that would put me off developing future plugins.

Adding something as simple as --destructiveChangesPre destructiveChangesPre.xml and --destructiveChangesPost destructiveChangesPost.xml is exactly what people are after and have been for some time, see - #188. This is what led me to write a quick plugin in the first place.

Without destructive changes support in source:deploy, people will still need to convert and use mdapi:deploy. I made a comment in that other issue at the time, this may be a small bit of functionality but it blocks use of source:deploy so all the other great work you've all been doing isn't going to get used by the people needing destructive changes (which I'm guessing will be most people), even fewer now that there is no longer a workaround.

I've worked on projects that do destructive changes in different ways:

• Our most commonly used simple approach is to maintain destructive changes files manually in source control that get pushed to all sandboxes with every deployment - need to turn warnings off on deploy so you don't get errors from trying to delete something that doesn't exist but it works. After the production deployment we clear down the destructive changes files to stop them growing for ever and ever.
• The more advanced approach that I used a while back was as you suggest, dynamically build the destructive changes based on a list of deleted files from a git diff (keeping track of environment state on branches).

Thanks for taking the time to reply, especially with so much information - it's really appreciated.

[gdman]

Important thing I forgot to mention on my last point of automating destructive changes using the deleted files of a git diff.

Destructive changes aren't as simple as working out which files have been deleted, also need to check for modified files where sub components have been removed e.g. workflow rules, email alerts, labels, sharing rules, etc. SFDX source format should make that easier for objects but doesn't decompose all other metadata types.

That's a key reason I've not moved away from maintaining destructive changes files manually for releases at the moment.

[lnoellemorgan-zz]

@gdman Thank you for your feedback on communications. We definitely try to be proactive about communicating these types of changes but understand not everyone regularly monitors this repo unless they have an issue they're trying get fixed. We have some work planned in the next major release to increase our communication of changes and new features via the command line. Thanks again! Your feedback is incredibly valuable.

[dclappert]

Found this thread via #188. To express some additional interest in what @gdman has already stated, our CICD pipeline deploys metadata via source:deploy, and builds a list of metadata components to be deployed by performing a git diff between the last successfully deployed commit and the head commit. We are now looking into what it would take to support destructive changes, as our project is in source format. Our hope is that we'd be able to validate a single package containing both destructive and non-destructive changes derived from performing a diff between commits, and building a list of components to deploy (new or modified metadata), and a list of components to delete (deleted or renamed metadata). At the moment, we manually perform pre-validation destructive changes. However this adds friction, is not repeatable, and imposes a degree of deployment risk. This becomes especially apparent in our lower environments that become temporarily non-compilable due to required pre-validation destructive changes. Being able to validate a single package using source:deploy would greatly improve the capabilities of our pipeline.

[mshanemc]

I'm closing this since we've shipped all the destructive changes stuff that y'all brought up.

Closing it does not imply that we fixed the "hooks changed" problem