source:convert issue with CLI V7.112.0

[avrilzen]

Summary

Failure deploying to a Generation 1 package org using CLI v7.112.0.
sfdx force:mdapi:deploy fails on Fatal Error UNKNOWN_EXCEPTION:

Before running the deploy sfdx force:source:convert - n 'packagename' was run.

Under v7.110 there is no problem with deployment. Had to uninstall cli and install a previous version for deployment to be successful.

Steps To Reproduce:

1 Make sure you have an aura component and are deploying a generation 1 package.
2 Run sfdx force:source:convert -d outputdir - n 'packagename'
3 sfdx force:mdapi:deploy -d outputdir/packagename -u packageorg

Not sure if this is significant but in sfdx-project.json our default path is not force-app.

Running convert on v7.112 the following differs from v7.110.

1. A subfolder with the -n packagename value is created. The components to be deployed and package.xml file are under this subfolder.
2. The package.xml file is half the size of that of v7.110.

Expected result

Deployment to a patch org or package org should have been successful. It is successful when using cli v7.110.

Actual result

Deployment FATAL ERROR: UNKNOWN_EXCEPTION: Unable to handle Aura Definition 'auracomponentname/auracomponentname/auracomponentname.design'. filepath not expected

System Information

Run sfdx version --verbose --json and paste the output here.
Information no longer available. I had to uninstall cli and install a previous version.

Additional information

Feel free to attach a screenshot.

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[mshanemc]

I made some edits for clarity--this is an issue with the new version of source:convert. What we've seen so far is that most errors are related to unusual folder structures and file locations.

for example, having empty Aura/LWC bundles, or bundles without the normal file types: #1099

or putting non-object-related files inside an object folder.

Is there anything unusual about our project's structure or that aura component specifically? If you're able to replicate with a simplified example, please share that (zip file, git repo, etc).

[avrilzen]

Thanks for quick response, I think you have identified the issue. The folder structure has two SelectTicketPlan folders instead of just one. The second one is a subfolder of the 1st one.

src-path
src-path-subfolder
aura
SelectTicketPlan
SelectTicketPlan

I'm happy to remove the extraneous one and see if the issue is resolved. Will update case with outcome.

[avrilzen]

I removed the extraneous folder. This solved the Aura Definition issue. However the deploy still fails with 584 errors. This represents missing types in the package.xml file namely Custom Field, List View, Record Type, Compact Layout and Button or Link, The v7.112.1 package.xml is 32K whereas the v7.110 is 64k.

Here is the sfdx-project-json file. We do not have force-app in the path.
{
"packageDirectories": [
{
"path": "pkgsrc",
"default": true
}

],

"sfdcLoginUrl": "https://login.salesforce.com",
"sourceApiVersion": "52.0"
}

Here is the version info I did not provide previously

{
"cliVersion": "sfdx-cli/7.112.1",
"architecture": "win32-x64",
"nodeVersion": "node-v14.17.4",
"pluginVersions": [
"@oclif/plugin-autocomplete 0.3.0 (core)",
"@oclif/plugin-commands 1.3.0 (core)",
"@oclif/plugin-help 3.2.2 (core)",
"@oclif/plugin-not-found 1.2.4 (core)",
"@oclif/plugin-plugins 1.10.1 (core)",
"@oclif/plugin-update 1.4.0-3 (core)",
"@oclif/plugin-warn-if-update-available 1.7.0 (core)",
"@oclif/plugin-which 1.0.3 (core)",
"@salesforce/sfdx-plugin-lwc-test 0.1.7 (core)",
"@salesforce/sfdx-trust 3.6.0 (core)",
"alias 1.1.10 (core)",
"apex 0.2.3 (core)",
"auth 1.7.1 (core)",
"config 1.2.23 (core)",
"custom-metadata 1.0.12 (core)",
"data 0.6.0 (core)",
"generator 1.1.8 (core)",
"limits 1.2.1 (core)",
"org 1.6.7 (core)",
"salesforce-alm 52.2.3 (core)",
"schema 1.0.8 (core)",
"sfdx-cli 7.112.1 (core)",
"source 1.0.6 (core)",
"telemetry 1.2.3 (core)",
"templates 52.1.0 (core)",
"user 1.4.0 (core)"
],
"osVersion": "Windows_NT 10.0.19042"
}

[edralph]

I'm seeing what I assume is the same issue.

Upon sfdx force:source:convert I also get these differences with my current CLI (7.113.0-1159219):

• the PackageName being used as the foldername underneath the outputdir
• the Package.xml has a lot of stuff missing

A comment above mentions unusual folder locations; however I always use the suggested default folder locations.

This was originally happening with CLI version 7.112.1 so I upgraded to the latest hoping it would be fixed but no.

The outcome of Package.xml having stuff missing is deployment failure.

Is the workaround to reinstall an older CLI? I had uninstalled and reinstalled sfdx due to another recent issue.

[shetzel]

This issue: "the PackageName being used as the foldername underneath the outputdir" is being fixed and will be included in the CLI release candidate that will be published later today.

This issue: "the Package.xml has a lot of stuff missing" we need more details. Can you diff the files and let us know what's missing? There is a different library being used for the convert and deploy so there will be differences. Some of those differences (even file size) can be ok and some are not. We need to know what the differences are though before we can address them.

As a workaround, the most reliable and easiest method is to install an older version of the CLI until the issue is fixed.

[shetzel]

@avrilzen - The types you listed (Custom Field, List View, Record Type, Compact Layout and Button or Link) are children of CustomObjects. What does that folder structure look like? They should all be directly under .../objects/<custom_object_name>/ e.g., for list views, .../objects/<custom_object_name>/listViews/<list_view_files>

[edralph]

Thanks for your comment @shetzel .

Issue 1: PackageName... Understood.

Issue 2: Package.xml stuff missing...
Note: I always use the default locations for project files / wherever sfdx force:source:pull puts them.

Missing components identified during the failed metadata deploy are:
Custom object ListViews (all of them)
Custom object fields
Custom Metadata Type fields

Installing older CLI
I've seen elsewhere that downgrading the CLI is a workaround for now - but when I try using sfdx plugins:install sfdx-cli@ it appears to have installed the older version, but then the actual version is the same as reported by sfdx version:

edralph@Eds-MacBook-Pro-2 SuperRoundRobin % sfdx version
sfdx-cli/7.113.0 darwin-x64 node-v14.17.1
edralph@Eds-MacBook-Pro-2 SuperRoundRobin % sfdx plugins:install [email protected]
This plugin is not digitally signed and its authenticity cannot be verified. Continue installation y/n?: y
Finished digital signature check.
warning sfdx-cli > salesforce-alm > [email protected]: request-promise-native has been deprecated because it extends the now deprecated request package, see request/request#3142
warning sfdx-cli > salesforce-alm > [email protected]: request has been deprecated, see request/request#3142
warning sfdx-cli > salesforce-alm > jsforce > [email protected]: request has been deprecated, see request/request#3142
warning sfdx-cli > salesforce-alm > @salesforce/source-deploy-retrieve > @salesforce/core > jsforce > [email protected]: request has been deprecated, see request/request#3142
warning sfdx-cli > salesforce-alm > request > [email protected]: this library is no longer supported
warning sfdx-cli > salesforce-alm > request > [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
warning sfdx-cli > @salesforce/plugin-generator > nps-utils > opn-cli > temp-write > [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
warning sfdx-cli > @salesforce/plugin-generator > yeoman-environment > globby > fast-glob > micromatch > snapdragon > source-map-resolve > [email protected]: https://github.com/lydell/resolve-url#deprecated
warning sfdx-cli > @salesforce/plugin-generator > yeoman-environment > globby > fast-glob > micromatch > snapdragon > source-map-resolve > [email protected]: Please see https://github.com/lydell/urix#deprecated
warning "sfdx-cli > @salesforce/plugin-generator > [email protected]" has unmet peer dependency "eslint@>=7.20.0".
warning "sfdx-cli > @salesforce/plugin-generator > [email protected]" has unmet peer dependency "eslint@>=7.20.0".
warning "sfdx-cli > @salesforce/plugin-generator > eslint-config-xo-space > [email protected]" has unmet peer dependency "eslint@>=7.20.0".
Installing plugin sfdx-cli... installed v7.110.0
edralph@Eds-MacBook-Pro-2 SuperRoundRobin % sfdx version
sfdx-cli/7.113.0 darwin-x64 node-v14.17.1
edralph@Eds-MacBook-Pro-2 SuperRoundRobin %

[avrilzen]

I am confirming that this is the folder structure for objects.
They should all be directly under .../objects/<custom_object_name>/ e.g., for list views, .../objects/<custom_object_name>/listViews/<list_view_files>

On examining the converted content of a specific object the types are present (listviews, fields, recordtypes, validation rules...). They just don't get added to the package.xml file. This then becomes a deployment issue.

[edralph]

Ditto @shetzel - folder structure for my objects is the same.

objects/Custom_Object__c/fields
objects/Custom_Object__c/listViews

These don't get added to the package.xml file and then get a metadata deploy failure because these listviews and fields are not in package.xml.

[shetzel]

@avrilzen @edralph - Are the CustomObject entries in package.xml there? The individual fields and list views don't need to be in the package.xml for a deploy unless you are deploying them individually. This is one of the differences between conversion in the old plugin versus the source plugin. If you are deploying the entire CustomObject it will only list that object in the package.xml and that should be fine. That could account for all of the size difference between the old and new conversion manifests.

When you say it causes deploy failures, what is the command you used (and all flags) and what are the errors? We have lots of tests that ensure this works and I just tried using the ebikes-lwc repo and it worked. The package.xml contents were as you described for CustomObjects but the deploy was successful and I see those fields and list views in the org.

[edralph]

@shetzel - yes the CustomObject entries are in package.xml. The only reason I mention that they are not there is the deploy error message says they are not in there.

I use this to deploy:

sfdx force:mdapi:deploy -d mdapi-source/latest-package/PackageName -u PkgOrg -l RunLocalTests -w 10

=== Component Failures [96] TYPE FILE NAME PROBLEM ───── ───────────────────────────────────────────────────────── ───────────────────────────────────────────────────── ────────────────── Error SuperRoundRobin/objects/AssigneeCredit__c.object AssigneeCredit__c.All Not in package.xml Error SuperRoundRobin/objects/AssigneeCredit__c.object AssigneeCredit__c.Assignee__c Not in package.xml

Another 90 or so lines of that... then

ERROR running force:mdapi:deploy: The metadata deploy operation failed.

And when I look at the Deployment Status in my packaging org, I see the same output:

[avrilzen]

@shetzel do your tests cover the case of deploying to a package org? This seems to be common to both of us on this thread experiencing the same issue.

[edralph]

@avrilzen, how did you install a previous version of the CLI? When I run sfdx plugins:install [email protected] it says it has installed this version, but then running sfdx version then reveals that it hasn't actually installed the old version.

[avrilzen]

@edralph Ditto re the sfdx plugins:install not being effective. I had to uninstall cli and then install a specific version using npm. npm install sfdx-cli@ –global. In VS Code I also had to set terminal.integrated.defaultProfile.windows. This may not be necessary in your case.

[avrilzen]

@edralph - npm on windows

[edralph]

@avrilzen I installed npm, uninstalled sfdx-cli and then installed the cli using npm. Worked a treat and now I'm back on 7.110.0 and my convert / deploy process works as it used to. Thanks!

[Szandor72]

I have source:convert --packagename 'package' failing on CircleCI too with the most recent version - not only on my windows machine. By failing I mean it fails to provide a complete package.xml containing all Custom fields and Objects and necessary entries for CustomLabels and Translations.

[shetzel]

@avrilzen - No the tests do not deploy to a packaging org. I'll mark this as a bug and get those fields back in.
@edralph - I know you got your answer already but to install a different version you can uninstall the current version and install via npm (as you did) or you can use a previous version of an installer or tar. See this release notes for where to find them for your platform. https://github.com/forcedotcom/cli/blob/main/releasenotes/README.md#51140-may-27-2021---cli-71030

[uip-robot-zz]

This issue has been linked to a new work item: W-9762933

[WillieRuemmele]

Hi everyone, this should be fixed in the latest-rc build of the cli 1.115.0, you can get it by either sfdx update stable-rc or npm install sfdx-cli@latest-rc --global depending on your preferred install method.

Please let us know if you're still seeing this in 1.115.0 of the CLI

[davisagli]

@WillieRuemmele I tested with the latest RC (7.115.1-a6551a9) and I am still seeing this problem where child components of CustomObject like CustomField are not included in package.xml when the force:source:convert command is run with the -n option.

[shetzel]

This issue has at least 3 sub-issues and not all of them are fixed.

Fixed:

1. the convert command when used with -n or --packagename now writes to the same location as with the salesforce-alm plugin
2. better error messages are displayed when metadata files are discovered in unexpected locations

Not Fixed:

1. convert of custom object children writes those entries to the manifest.

This last one is something I hope to get into the next CLI release candidate.

[avrilzen]

@mshanemc @shetzel I updated cli to v7.115.1 to test the fix for deploying to a packaging org.
{
"cliVersion": "sfdx-cli/7.115.1",
"architecture": "win32-x64",
"nodeVersion": "node-v14.17.4",
"pluginVersions": [
"@oclif/plugin-autocomplete 0.3.0 (core)",
"@oclif/plugin-commands 1.3.0 (core)",
"@oclif/plugin-help 3.2.3 (core)",
"@oclif/plugin-not-found 1.2.4 (core)",
"@oclif/plugin-plugins 1.10.1 (core)",
"@oclif/plugin-update 1.4.0-3 (core)",
"@oclif/plugin-warn-if-update-available 1.7.0 (core)",
"@oclif/plugin-which 1.0.3 (core)",
"@salesforce/sfdx-plugin-lwc-test 0.1.7 (core)",
"@salesforce/sfdx-trust 3.6.0 (core)",
"alias 1.1.10 (core)",
"apex 0.2.7 (core)",
"auth 1.7.1 (core)",
"config 1.2.24 (core)",
"custom-metadata 1.0.12 (core)",
"data 0.6.0 (core)",
"generator 1.2.0 (core)",
"limits 1.2.1 (core)",
"org 1.7.0 (core)",
"salesforce-alm 52.2.6 (core)",
"schema 1.0.8 (core)",
"sfdx-cli 7.115.1 (core)",
"source 1.0.10 (core)",
"telemetry 1.2.3 (core)",
"templates 52.1.0 (core)",
"user 1.4.0 (core)"
],
"osVersion": "Windows_NT 10.0.19042"
}

A sfdx force:source:convert -d .mdapioutput -n “packageName” was run.

While the missing types reported previously have now been included in the package.xml file, the deploy is still failing on missing types in the package.xml. In my case the missing types are Workflow Rule and Workflow Field Update.

I added these types manually to the generated package.xml file. On doing this, the deploy to the packaging org is successful.

I think this ticket is in closed status. Can it be reopened or does a new ticket need to be created? Thanks

[shetzel]

@avrilzen - Sorry the fix is incomplete. That should have fixed it for all child types but Workflow looks like it's categorized incorrectly so the child types are not added. I'll reopen this and work on another fix.