aws-cdk: Asset publishing fails randomly

[toha-nordic]

Describe the bug

We have a system with 10+ stacks in a single App. To speed up deployments, we have started to use parallel deployments but it fails randomly.

Running cdk deploy --all --concurrency 10 --asset-parallelism --asset-prebuild=true fails to publish assets for some (random) stacks every now and then (fail: ENOENT: no such file or directory).

We have tried also without --asset-prebuild=true but that doesn't help.
Not sure if its any way related but please notice how some lines have current_account-current_region at the end, and not the real AWS account + region?

Any help or fixes would be much appreciated!

Expected Behavior

All assets are published successfully and all stacks are deployed.

Current Behavior

[40%] success: Published 663e668fa203832f3fcf463ff3ead687309ffb0619ab9559be58be0f03b62081:current_account-current_region
[60%] fail: ENOENT: no such file or directory, unlink '/codebuild/output/src192604038/src/github.com/<repo path hidden>/cdk.out/.cache/903be65b9594a9f17d7120273620367e7830b6d087fc9591f31dc5a080049e46.zip'
[60%] fail: ENOENT: no such file or directory, unlink '/codebuild/output/src192604038/src/github.com/<repo path hidden>/cdk.out/.cache/903be65b9594a9f17d7120273620367e7830b6d087fc9591f31dc5a080049e46.zip'
[100%] success: Published 6b5dc596ec690dde29f75b9582cad05d4805cee1eba67ded6ee62d886dae809c:current_account-current_region
[80%] success: Published 49326c4cc1d74b27b77764faee05b791dbb43e5459119175d2af4868de4b14c0:current_account-current_region
[52%] success: Published bcd01ef38be2d18c8ac80b92f52f86ede3deedbe88226d037a71fa8aa2889571:<AWS account hidden>-us-east-1
[80%] success: Published 083fbff4b4cbef7ebed12ee1abbbbbe251b8ccd2d4ae995638ac3585dc2101ff:current_account-current_region
[60%] success: Published 848cda52ce29d931a6537ae9543a357350c0e792bc49b6c9cb496ffc9552c297:current_account-current_region
[100%] success: Published 0380c831fe118faed624d412aa3a52646a32802277d9f2ad0b03d3514eb4ff90:current_account-current_region
 ❌  <my stack name here> failed: Error: Failed to publish one or more assets. See the error messages above for more information.
    at publishAssets (/codebuild/output/src192604038/src/github.com/<repo path hidden>/node_modules/aws-cdk/lib/index.js:374:73647)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async CloudFormationDeployments.publishStackAssets (/codebuild/output/src192604038/src/github.com/<repo path hidden>/node_modules/aws-cdk/lib/index.js:381:87962)
    at async CloudFormationDeployments.deployStack (/codebuild/output/src192604038/src/github.com/<repo path hidden>/node_modules/aws-cdk/lib/index.js:381:84233)
    at async deployStack2 (/codebuild/output/src192604038/src/github.com/<repo path hidden>/node_modules/aws-cdk/lib/index.js:383:145458)
    at async /codebuild/output/src192604038/src/github.com/<repo path hidden>/node_modules/aws-cdk/lib/index.js:383:128776
    at async run (/codebuild/output/src192604038/src/github.com/<repo path hidden>/node_modules/aws-cdk/lib/index.js:383:126782)

Reproduction Steps

Unfortunately not sure what triggers this, and how to reliably reproduce it.
At least we have seen it with an App which contains several Stacks with lots of AWS cloud resources.
Turning --concurrency off fixes this issue.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.72.0 (build 397e46c)

Framework Version

No response

Node.js Version

v18.15.0

OS

Linux Ubuntu 20/22.04, https://github.com/aws/aws-codebuild-docker-images/tree/master/ubuntu/standard/6.0

Language

Typescript

Language Version

4.7.3

Other information

Similar case was described in #24298. #24298 (comment) mentioned of a possible fix in CDK v2.64.0 but we still have the issue even with v2.67.0 and v2.72.0.

[pahud]

Thank you for your report. We will bring this up to the core team maintainers and see if there's anything we could do for to address this.

[corymhall]

I'm going to close this in favor of #25380 since that one has a linked pr.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.