Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sfdx force:source:push failing since the latest stable update 7.144.2 #1466

Closed
sesh-kebab opened this issue Apr 4, 2022 · 21 comments
Closed
Labels
more information required Issue requires more information or a response from the customer

Comments

@sesh-kebab
Copy link

Summary

Latest sfdx-cli, 7.144.2 has broken pushes for our codebase.

Steps To Reproduce:

Unable to provide a sample repository but when push was working it would push approximately 7,600 components (between two packages 7,400 and 200 in another) files along with a static resource directly that was greater than 50mb to give you an idea of approximate size that this issue is reproducible in.

Create a new scratch org and set as default scratch org.

  1. sfdx force:source:push
*** Pushing v52.0 metadata with SOAP API v53.0 connection ***
ERROR running force:source:push:  INVALID_OPERATION: Too many files in zip
  1. sfdx force:config:set restDeploy=true
  2. sfdx force:source:push
*** Pushing v52.0 metadata with REST API v53.0 connection ***
ERROR running force:source:push:  Metadata API request failed: The deploy operation is missing a job ID. Initialize an operation with an ID, or start a new job.
  1. Revert restDeploy: sfdx force:config:set restDeploy=false
  2. Add "pushPackageDirectoriesSequentially": true, to sfdx-project.json
  3. sfdx force:source:push (note adding pushPackageDirectoriesSequentially has no effect). We have two packages in a single namespace so I would have expected this to make a difference if it works as intended
*** Pushing v52.0 metadata with SOAP API v53.0 connection ***
Error: write EPIPE
  at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16) {
errno: -32,
code: 'EPIPE',
syscall: 'write'
}
ERROR running force:source:push:  write EPIPE
  1. sfdx force:source:legacy:push
*** Deploying with SOAP ***
Job ID | 0Af1D00001vYkVJSA0
Job ID | 0Af1D00001vYkVJSA0
SOURCE PROGRESS | ████████████████████████████████████████ | 7411/7411 Components
*** Deploying with SOAP ***
Job ID | 0Af1D00001vYnD1SAK
SOURCE PROGRESS | ████████████████████████████████████████ | 211/213 Components
...

Expected result

Source for multiple packages should be pushed without using legacy.

Actual result

Failed with "Too many files in zip" error. Attempting to use workarounds of:

  1. using RestDeploy
  2. or new pushPackageDirectoriesSequentially attribute

both failed with different set of errors as described above.

System Information

sfdx version --verbose --json
{
	"cliVersion": "sfdx-cli/7.144.2",
	"architecture": "darwin-x64",
	"nodeVersion": "node-v16.14.2",
	"pluginVersions": [
		"@oclif/plugin-autocomplete 0.3.0 (core)",
		"@oclif/plugin-commands 1.3.0 (core)",
		"@oclif/plugin-help 3.3.1 (core)",
		"@oclif/plugin-not-found 1.2.6 (core)",
		"@oclif/plugin-plugins 1.10.11 (core)",
		"@oclif/plugin-update 1.5.0 (core)",
		"@oclif/plugin-warn-if-update-available 1.7.3 (core)",
		"@oclif/plugin-which 1.0.4 (core)",
		"@salesforce/lwc-dev-server 2.11.0",
		"@salesforce/sfdx-plugin-lwc-test 0.1.7 (core)",
		"@salesforce/sfdx-scanner 2.13.1",
		"alias 1.2.1 (core)",
		"apex 0.11.0 (core)",
		"auth 1.8.1 (core)",
		"community 1.1.4 (core)",
		"config 1.3.23 (core)",
		"custom-metadata 1.0.12 (core)",
		"custom-plugin-test 0.0.0 (link) /Users/schemudugunta/dev/sfdx-playground/custom-plugin-test",
		"data 0.6.11 (core)",
		"generator 1.2.2 (core)",
		"info 1.3.1 (core)",
		"limits 1.3.0 (core)",
		"org 1.11.2 (core)",
		"salesforce-alm 54.1.0 (core)",
		"schema 1.1.0 (core)",
		"sfdx-cli 7.144.2 (core)",
		"source 1.9.2 (core)",
		"telemetry 1.4.0 (core)",
		"templates 54.3.0 (core)",
		"trust 1.1.0 (core)",
		"user 1.7.1 (core)"
	],
	"osVersion": "Darwin 21.3.0"
}
@sesh-kebab sesh-kebab added the investigating We're actively investigating this issue label Apr 4, 2022
@github-actions
Copy link

github-actions bot commented Apr 4, 2022

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

@WillieRuemmele
Copy link
Member

Hi @sesh-kebab 👋

thanks for reporting this. I just want to verify that you're setting pushPackageDirectoriesSequentially as described here. It should be picking it up and pushing as two separate operations.

@sesh-kebab
Copy link
Author

Hi @WillieRuemmele , correct using it as described, by adding it to sfdx-project.json.

@WillieRuemmele
Copy link
Member

ok, is this happening every time you've ran the source:push command? Some of the errors in your message point to connectivity issues - and so running the commands multiple times might get a stable enough connection to push successfully. If you're unable to get it to connect, can you append --dev-debug to your push command and paste the output?

@sesh-kebab
Copy link
Author

Sure, i'll give that a try. But a couple of other data points if they might help:

  • the push breaking behavior was also introduced just prior to 7.141.1March 10, 2022 release and was subsequently fixed in this release with the breaking changes having been reverted
  • the connection from my laptop, especially on VPN is not the best at times, but we do see this consistently in our ci/cd pipelines too (pipeline runners run on ec2 instances in aws).

@sesh-kebab
Copy link
Author

Additional excerpts with --dev-debug enabled

sfdx force:source:push (SOAP)
sfdx:AuthInfo INFO Updated auth info for username: [email protected] +11s
  sfdx:AuthInfo INFO Returning fields for a connection using OAuth config. +2ms
  sfdx:core TRACE Setup child 'connection' logger instance +98ms
  sfdx:connection DEBUG Last checked on 4/4/2022, 10:19:04 AM (now is 4/4/2022, 10:55:19 PM) - not getting latest +11s
  sfdx:connection DEBUG Loaded latest apiVersion 54.0 +1ms
  sfdx:connection DEBUG Using apiVersion 54.0 +0ms
  sfdx:MetadataApiDeploy DEBUG Overriding apiVersion to: 53.0 +0ms
  sfdx:connection DEBUG deploy with SOAP +0ms
ERROR running force:source:push:  INVALID_OPERATION: Too many files in zip 
*** Internal Diagnostic ***

sf:INVALID_OPERATION: INVALID_OPERATION: Too many files in zip
    at module.exports.HttpApi.getError (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/jsforce/lib/http-api.js:250:13)
    at /Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/jsforce/lib/http-api.js:95:22
    at tryCallOne (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/promise/lib/core.js:37:12)
    at /Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/promise/lib/core.js:123:15
    at flush (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/asap/raw.js:50:29)
    at processTicksAndRejections (node:internal/process/task_queues:78:11)
Outer stack:
    at Function.wrap (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/core/lib/sfdxError.js:171:27)
    at Push.catch (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/command/lib/sfdxCommand.js:248:67)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Push._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/command/lib/sfdxCommand.js:85:13)
    at async Config.runCommand (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/config/lib/config.js:173:24)
    at async SfdxMain.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/main.js:27:9)
    at async SfdxMain._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/command.js:43:20)
    at async Object.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/dist/cli.js:162:47)
******

  sfdx:Push ERROR [
  sfdx:Push   '\x1B[1mERROR running force:source:push: \x1B[22m',
  sfdx:Push   '\x1B[31mINVALID_OPERATION: Too many files in zip\x1B[39m',
  sfdx:Push   '\x1B[31m\x1B[39m\n' +
  sfdx:Push     '\x1B[31m*** Internal Diagnostic ***\x1B[39m\n' +
  sfdx:Push     '\x1B[31m\x1B[39m\n' +
  sfdx:Push     '\x1B[31msf:INVALID_OPERATION: INVALID_OPERATION: Too many files in zip\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at module.exports.HttpApi.getError (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/jsforce/lib/http-api.js:250:13)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at /Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/jsforce/lib/http-api.js:95:22\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at tryCallOne (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/promise/lib/core.js:37:12)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at /Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/promise/lib/core.js:123:15\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at flush (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/asap/raw.js:50:29)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at processTicksAndRejections (node:internal/process/task_queues:78:11)\x1B[39m\n' +
  sfdx:Push     '\x1B[31mOuter stack:\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at Function.wrap (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/core/lib/sfdxError.js:171:27)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at Push.catch (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/command/lib/sfdxCommand.js:248:67)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at processTicksAndRejections (node:internal/process/task_queues:96:5)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async Push._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/command/lib/sfdxCommand.js:85:13)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async Config.runCommand (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/config/lib/config.js:173:24)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async SfdxMain.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/main.js:27:9)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async SfdxMain._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/command.js:43:20)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async Object.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/dist/cli.js:162:47)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m******\x1B[39m\n' +
  sfdx:Push     '\x1B[31m\x1B[39m'
  sfdx:Push ] +1m
  sfdx:core TRACE Setup child 'Org' logger instance +1m
  sfdx:core TRACE Setup child 'Aliases' logger instance +1ms
sfdx force:source:push (REST)
sfdx:MetadataApiDeploy DEBUG Overriding apiVersion to: 53.0 +0ms
  sfdx:connection DEBUG deploy with REST +1ms
  sfdx:connection DEBUG Refreshing auth for org. +1ms
  sfdx:connection DEBUG request: {"url":"https://customer-business-5099-dev-ed.cs44.my.salesforce.com/services/data/v53.0","method":"GET","headers":{"content-type":"application/json","user-agent":"sfdx toolbelt:"}} +0ms
  sfdx:MetadataApiDeploy DEBUG Started metadata transfer. ID = undefined +37s
  sfdx:Push DEBUG initializing progress bar +46s
  sfdx:core TRACE Setup child 'PollingClient' logger instance +37s
  sfdx:MetadataApiDeploy DEBUG Polling for metadata transfer status. ID = undefined +92ms
  sfdx:MetadataApiDeploy DEBUG Polling frequency (ms): 5710 +0ms
  sfdx:MetadataApiDeploy DEBUG Polling timeout (min): 33 +0ms
  sfdx:MetadataApiDeploy ERROR  +5ms
ERROR running force:source:push:  Metadata API request failed: The deploy operation is missing a job ID. Initialize an operation with an ID, or start a new job. 
*** Internal Diagnostic ***

MetadataTransferError: Metadata API request failed: The deploy operation is missing a job ID. Initialize an operation with an ID, or start a new job.
    at MetadataApiDeploy.pollStatus (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/source-deploy-retrieve/lib/src/client/metadataTransfer.js:83:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Push.deploy (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/plugin-source/lib/commands/force/source/push.js:80:28)
    at async Push.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/plugin-source/lib/commands/force/source/push.js:29:9)
    at async Push._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/command/lib/sfdxCommand.js:81:40)
    at async Config.runCommand (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/config/lib/config.js:173:24)
    at async SfdxMain.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/main.js:27:9)
    at async SfdxMain._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/command.js:43:20)
    at async Object.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/dist/cli.js:162:47)
DUE TO:
MissingJobIdError: The deploy operation is missing a job ID. Initialize an operation with an ID, or start a new job.
    at MetadataApiDeploy.checkStatus (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/source-deploy-retrieve/lib/src/client/metadataApiDeploy.js:245:19)
    at MetadataApiDeploy.poll (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/source-deploy-retrieve/lib/src/client/metadataTransfer.js:155:42)
    at doPoll (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/core/lib/status/pollingClient.js:62:45)
    at /Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:90:48
    at /Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:138:42
    at step (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:46:23)
    at Object.next (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:27:53)
    at /Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:21:71
    at new Promise (<anonymous>)
    at __awaiter (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:17:12)
Outer stack:
    at Function.wrap (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/core/lib/sfdxError.js:171:27)
    at Push.catch (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/command/lib/sfdxCommand.js:248:67)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Push._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/command/lib/sfdxCommand.js:85:13)
    at async Config.runCommand (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/config/lib/config.js:173:24)
    at async SfdxMain.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/main.js:27:9)
    at async SfdxMain._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/command.js:43:20)
    at async Object.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/dist/cli.js:162:47)
******

  sfdx:Push ERROR [
  sfdx:Push   '\x1B[1mERROR running force:source:push: \x1B[22m',
  sfdx:Push   '\x1B[31mMetadata API request failed: The deploy operation is missing a job ID. Initialize an operation with an ID, or start a new job.\x1B[39m',
  sfdx:Push   '\x1B[31m\x1B[39m\n' +
  sfdx:Push     '\x1B[31m*** Internal Diagnostic ***\x1B[39m\n' +
  sfdx:Push     '\x1B[31m\x1B[39m\n' +
  sfdx:Push     '\x1B[31mMetadataTransferError: Metadata API request failed: The deploy operation is missing a job ID. Initialize an operation with an ID, or start a new job.\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at MetadataApiDeploy.pollStatus (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/source-deploy-retrieve/lib/src/client/metadataTransfer.js:83:27)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at processTicksAndRejections (node:internal/process/task_queues:96:5)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async Push.deploy (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/plugin-source/lib/commands/force/source/push.js:80:28)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async Push.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/plugin-source/lib/commands/force/source/push.js:29:9)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async Push._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/command/lib/sfdxCommand.js:81:40)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async Config.runCommand (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/config/lib/config.js:173:24)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async SfdxMain.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/main.js:27:9)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async SfdxMain._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/command.js:43:20)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async Object.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/dist/cli.js:162:47)\x1B[39m\n' +
  sfdx:Push     '\x1B[31mDUE TO:\x1B[39m\n' +
  sfdx:Push     '\x1B[31mMissingJobIdError: The deploy operation is missing a job ID. Initialize an operation with an ID, or start a new job.\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at MetadataApiDeploy.checkStatus (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/source-deploy-retrieve/lib/src/client/metadataApiDeploy.js:245:19)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at MetadataApiDeploy.poll (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/source-deploy-retrieve/lib/src/client/metadataTransfer.js:155:42)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at doPoll (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/core/lib/status/pollingClient.js:62:45)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at /Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:90:48\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at /Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:138:42\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at step (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:46:23)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at Object.next (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:27:53)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at /Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:21:71\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at new Promise (<anonymous>)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at __awaiter (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/ts-retry-promise/dist/retry-promise.js:17:12)\x1B[39m\n' +
  sfdx:Push     '\x1B[31mOuter stack:\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at Function.wrap (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/core/lib/sfdxError.js:171:27)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at Push.catch (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/command/lib/sfdxCommand.js:248:67)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at processTicksAndRejections (node:internal/process/task_queues:96:5)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async Push._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@salesforce/command/lib/sfdxCommand.js:85:13)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async Config.runCommand (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/config/lib/config.js:173:24)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async SfdxMain.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/main.js:27:9)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async SfdxMain._run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/node_modules/@oclif/command/lib/command.js:43:20)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m    at async Object.run (/Users/schemudugunta/.local/share/sfdx/client/7.144.2-2fa8aca/dist/cli.js:162:47)\x1B[39m\n' +
  sfdx:Push     '\x1B[31m******\x1B[39m\n' +
  sfdx:Push     '\x1B[31m\x1B[39m'
  sfdx:Push ] +111ms
  sfdx:core TRACE Setup child 'Org' logger instance +22ms
  sfdx:core TRACE Setup child 'Aliases' logger instance +1ms
  sfdx:Aliases INFO Reading config file: /Users/schemudugunta/.sfdx/alias.json +51s
  @oclif/config start postrun hook +46s
  @oclif/config postrun hook done +3ms
  sfdx:core TRACE Setup child 'AuthInfo' logger instance +14ms
  sfdx:core TRACE Setup child 'crypto' logger instance +0ms
  sfdx:crypto DEBUG retryStatus: undefined +37s
  sfdx:core TRACE Setup child 'keyChain' logger instance +1ms
  sfdx:keyChain DEBUG platform: darwin +37s
  sfdx:core TRACE Setup child 'AuthInfoConfig' logger instance +31ms
  sfdx:AuthInfoConfig INFO Reading config file: /Users/schemudugunta/.sfdx/[email protected] +52s
  sfdx:AuthInfo INFO Updated auth info for username: [email protected] +37s
  sfdx:AuthInfo INFO Returning fields for a connection using OAuth config. +1ms
  sfdx:core TRACE Setup child 'connection' logger instance +4ms
  sfdx:connection DEBUG Last checked on 4/4/2022, 1:01:51 AM (now is 4/4/2022, 11:11:28 PM) - not getting latest +37s
  sfdx:connection DEBUG Loaded latest apiVersion 54.0 +0ms
  sfdx:connection DEBUG Using apiVersion 54.0 +0ms
  sfdx:connection DEBUG request: {"method":"GET","url":"https://iqvia-17d-dev-ed.my.salesforce.com/services/data/v54.0/query?q=SELECT%20CreatedDate%2CEdition%2CExpirationDate%20FROM%20ActiveScratchOrg%20WHERE%20ScratchOrg%3D'00D7A0000000i5d'","headers":{"content-type":"application/json","user-agent":"sfdx toolbelt:"}} +1ms

@WillieRuemmele
Copy link
Member

I'm having a hard time reproducing this. I created exactly 7,411 ApexClasses in the first package, and 213 in the second to mimic the size of your deployment and both commands failed with the Too many files in zip error.

 ➜  sfdx force:source:legacy:push -u legacy
*** Deploying with SOAP ***
ERROR running force:source:legacy:push:  INVALID_OPERATION: Too many files in zip

and

 ➜  sfdx force:source:push -u beta                   
*** Pushing v45.0 metadata with SOAP API v53.0 connection ***
ERROR running force:source:push:  INVALID_OPERATION: Too many files in zip

I then reduced the ApexClasses by 2 (because there are 2 files for each) so that maybe the zip files would be similar to your size and source:push succeeded for me. As another workaround you could try using source:deploy --tracksource to deploy your project, and still include source tracking information so that push/pull will continue to work as expected

@shetzel shetzel added more information required Issue requires more information or a response from the customer and removed investigating We're actively investigating this issue labels Apr 11, 2022
@sesh-kebab
Copy link
Author

Our static resource folder is also around 45mb in size with 208 files. Not sure if the could also be a contributing factor?

find ./force-app/main/default -type f | wc -l
208

du -hd 1 ./force-app/main/default
...
45M    ./force-app/main/default/staticresources

Thanks for suggesting the workaround of source:deploy --tracksource. I'm able to proceed for now using sfdx force:source:legacy:push/pull. I'll try out the source:deploy --tracksource and report back.

@no-response no-response bot removed the more information required Issue requires more information or a response from the customer label Apr 11, 2022
@shetzel
Copy link
Contributor

shetzel commented Apr 15, 2022

@sesh-kebab - can you please set the SFDX_MDAPI_TEMP_DIR env var to some temp dir, then run both versions of the push commands, then compare the difference between the 2 sub-directories within the temp dir. Both those commands use the metadata API in the end so there must be a difference in the generated files/zips. Note that you'll have to clear all tracking data between the 2 command runs. Please report back with anything that can help us narrow this down. The static resources observation is a good one.

@shetzel shetzel added the more information required Issue requires more information or a response from the customer label Apr 15, 2022
@LionelSchroder
Copy link

I have similar issue but without too many files error message. Push with new CLI version is not working anymore while legacy push is working. I push all metadata in one push, 4700 components and size is around 40 mb but no big static resource folder in my case

Lot's of errors related to missing fields and class error compilation. In deploy UI seems that the first error causing the others is this one
CaseCustom Object 1866530 Error parsing file: The entity name must immediately follow the '&' in the entity reference

I'll try the above procedure to try to give more info

@sesh-kebab
Copy link
Author

@WillieRuemmele I had a chance to try the source:deploy --tracksource but oddly i'm seeing a lot of unexpected errors that I haven't seen before when deploying

sfdx force:source:deploy --tracksource (output obfuscated)
sfdx force:source:deploy -p OMR/force-app -u deploy --tracksource
*** Deploying v52.0 metadata with SOAP API v53.0 connection ***
(node:79342) Warning: The SFDX_MDAPI_TEMP_DIR environment variable is set, which may degrade performance
(Use `node --trace-warnings ...` to show where the warning was created)
Deploy ID: 0Af1D00001x9oTbSAI
DEPLOY PROGRESS | ████████████████████████████████████████ | 7827/7827 Components
Updating source tracking... done

=== Component Failures [63]
Type   Name                             Problem
─────  ───────────────────────────────  ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Error  <obfuscate>ListViewAll           Invalid definition for <obfuscate>fo: ApexService.getType() return null with currentNamespace: null, namespace: <obfuscate>, name: XXSObjectInfo
Error  dsPage                           We couldn't retrieve the design time component information for component <obfuscate>dsPage.
Error  <obfuscate>_Record_Page          We couldn't retrieve the design time component information for component <obfuscate>d.
Error  <obfuscate>_Record_Page          We couldn't retrieve the design time component information for component <obfuscate>ccountsDiffList.
Error  <obfuscate>_Record_Page          We couldn't retrieve the design time component information for component <obfuscate>yHighlightPanel.
Error  <obfuscate>_Record_Page          We couldn't retrieve the design time component information for component <obfuscate>ighlightPanel.
Error  <obfuscate>_Record_Page          We couldn't retrieve the design time component information for component <obfuscate>HighlightPanel.
Error  <obfuscate>_Record_Page          We couldn't retrieve the design time component information for component <obfuscate>HighlightPanel.
Error  <obfuscate>_Record_Page          We couldn't retrieve the design time component information for component <obfuscate>yHighlightPanel.
Error  map                              No base file for markup://<obfuscate>:map
Error  adjGrid                          duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  adj                              No base file for markup://<obfuscate>:adj
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  map                              duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  modal                            duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  xxxReportList  duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  xxxReportList  duplicate value found: <unknown> duplicates value on record with id: <unknown>

ERROR running force:source:deploy:  Deploy failed.

Seems similar to what @LionelSchroder is seeing (although I'm not sure what deploy UI is?)

@shetzel trying the legacy push and normal push with the temp dir overriden, I'm seeing the following

image

And when I drill into each of the respective metadata folders, I'm seeing
image

Apart from the .customIndex metadata folder when not using legacy, and the differently named metadata packages, I don't see any obvious differentiating factor from a size perspective that could explain the issue we're seeing.

Checking the number of files however, I'm seeing 218 additional files when not using legacy
image

I was examining what the additional files could have been when I noticed the sdx_mdpkg_1650### folder in the legacy mode, changed and now only contained 233 files. Looking at cli output, it was during the time that 1 out of 2 package directories was successfully pushed. So this (almost) explains the discrepancy between the file count. When using legacy, it seems the metadata temp folder is refreshed for each package directory, described in sfdx-project.json. When not using legacy, it seems all package directories are combined into a single metadata folder and it goes over a certain threshold before a push is attempted, resulting the in the "too many files" error message.

To rule out any issue with multiple package directories, I used a different sfdx-project.json which only included a single package directory, so that the metadata temp folders would be almost identical in size and file count; that still resulted in "ERROR running force:source:push: INVALID_OPERATION: Too many files in zip" message.

Does this help?

@LionelSchroder
Copy link

LionelSchroder commented Apr 20, 2022

First time I had the issue, as there is too much errors the output in my terminal was truncated. In UI the order of the errors is not the same as the one in the terminal. I was wondering if first error in UI could be at the origin of the others.

That's what I was talking about :
Capture d’écran 2022-04-20 à 15 56 59

@dyytina
Copy link

dyytina commented Apr 20, 2022

@shetzel I work with Seshi in the same project. I tried your suggestion by setting the SFDX_MDAPI_TEMP_DIR env var to some temp dir, then run push commands in two versions. I use old version V7.143.0 and new version V7.146.1 for comparison.
metadataPackages
FolderSize
I notice that in the older version, it created a sdx_decomposition folder with objects and staticresources but in the new version there is just one big folder includes everything.
Regarding the number of files, files generated from latest version of push has 7,342 items (57.2 MB on disk). Files generated from older version of push has 199 items(4.8 MB on disk) in sdx_decomposition sub folder and 7,344 items (57.9 MB on disk) in sdx_mdpkg sub folder. In both cases, it does not exceed the limit which is 10,000 files and 39MB compressed size.
Hope it helps.

@mshanemc
Copy link
Contributor

@LionelSchroder @dyytina I'm picking up this investigation.

I have a theory: can one of you share your .forceignore file? I'm wondering if there's some different behavior between the 2 versions. We do have a bug open about how ignore and profiles aren't quite right, and I do see profiles being 3x bigger in @sesh-kebab 's diff.

you can also run the force:source:ignored:list command that will tell you what the ignore file matches...so hopefully you can validate that it's doing what you think it should.

@dyytina
Copy link

dyytina commented Apr 21, 2022

Here is the screenshot of .forceignore file
Screen Shot 2022-04-21 at 11 45 43 AM

@dyytina
Copy link

dyytina commented May 2, 2022

Please let me know if you need more information on this ticket. Thank you!

@sesh-kebab
Copy link
Author

sesh-kebab commented May 5, 2022

seems to be stuck with more information required - commenting to see if it will trigger the label being removed.

@mshanemc please let us know if you need anything more? the issues in the original description are still present as of 7.147.1 sfdx cli.

@LionelSchroder
Copy link

My error is solved with 7.149.1

@sesh-kebab
Copy link
Author

@LionelSchroder unfortunately I'm not having the same luck. Although the errors about size are no longer present with 7.149.1 , I'm seeing a new set of errors:

=== Component Failures [4]
Type   Name                             Problem
─────  ───────────────────────────────  ──────────────────────────────────────────────────────────────────────────────
Error  omDashboardLinkWorker            duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  omDiscreteModalHlp               duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  omrGeographyTerritoryReportList  duplicate value found: <unknown> duplicates value on record with id: <unknown>
Error  omrGeographyTerritoryReportList  duplicate value found: <unknown> duplicates value on record with id: <unknown>

Not sure why the first two 'omDashboardLinkWorker' & 'omDiscreteModalHlp' could be throwing this error. But the second two are not duplicate LWCs but identically named components in different folders. That could be why but this wasn't causing an issue previously and can confirm using sfdx force:source:legacy:push works fine.

@mshanemc
Copy link
Contributor

mshanemc commented Jun 2, 2022

@sesh-kebab if you've got the same-named component in two different folders, legacy:push would do one folder at a time (last deploy would overwrite the first one).

You can modify your project.json to make regular push behave the name way (see that sequential property) to see if that solves it.

[and then decide if you really mean to have identically named components because that's weird and likely to cause other problems with merging/tracking since the server doesn't know about your local folders]

@sesh-kebab
Copy link
Author

Thanks @mshanemc, all. I'll close this case as I'm no longer working on this and will leave it up to the other participants to create a new case and link back to this case if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
more information required Issue requires more information or a response from the customer
Projects
None yet
Development

No branches or pull requests

6 participants