sf project deploy start --metadata --test-level RunSpecifiedTests --tests doesn't read the --tests when running --dry-run

[junners]

Summary

Running `sf project deploy start --manifest package.xml --test-level RunSpecifiedTests --tests "comma,separated,testsclasses" --dry-run doesn't read the value specified in "tests"

Steps To Reproduce:

1. sf project generate manifest --source-dir force-app
2. sf project deploy start --manifest package.xml --test-level RunSpecifiedTest
   s --dry-run --target-org --tests "FileUtilitiesTest,GeoCodingServiceTest,TestPropertyController,TestSampleDataController" --ignore-warnings

Notice that the dry run will fail because the tests levels aren't passing

*** Deploying (dry-run) v57.0 metadata with REST API v58.0 connection ***
Deploy ID:XXXXXX
Status: Failed | ████████████████████████████████████████ | 91/91 Components

Test Results Summary
Passing: 0
Failing: 0
Total: 0
Dry-run complete.

Expected result

Dry run should succeed since the classes indicated has enough coverage for the apex classes

Actual result

The dry-run deployment failed and did not read the specified tests, checking the org it says code coverage failure

System Information

• Which shell/terminal are you using? (e.g. bash, zsh, powershell 5, powershell 7, cmd.exe, etc.).
  zsh

• If you are using sfdx

  • Run sfdx version --verbose --json

• If you are using sf

  • Run sf version --verbose --json

• Paste the output here

sf cli
› Warning: @salesforce/cli update available from 1.76.1 to 1.76.2.

{
  "cliVersion": "@salesforce/cli/1.76.1",
  "architecture": "darwin-x64",
  "nodeVersion": "node-v18.15.0",
  "osVersion": "Darwin 21.6.0",
  "shell": "zsh",
  "rootPath": "/Users/jun.jose/.local/share/sfdx/client/7.198.7-cad6700/sf",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 2.1.9 (core)",
    "@oclif/plugin-commands 2.2.14 (core)",
    "@oclif/plugin-help 5.2.9 (core)",
    "@oclif/plugin-not-found 2.3.23 (core)",
    "@oclif/plugin-plugins 2.4.6 (core)",
    "@oclif/plugin-search 0.0.16 (core)",
    "@oclif/plugin-update 3.1.12 (core)",
    "@oclif/plugin-version 1.3.3 (core)",
    "@oclif/plugin-warn-if-update-available 2.0.35 (core)",
    "@oclif/plugin-which 2.2.19 (core)",
    "@salesforce/cli 1.76.1 (core)",
    "apex 2.2.13 (core)",
    "auth 2.7.12 (core)",
    "data 2.3.10 (core)",
    "deploy-retrieve 1.8.15 (core)",
    "dev 0.7.0 (user)",
    "env 2.1.8 (user)",
    "info 2.6.7 (core)",
    "limits 2.3.13 (core)",
    "login 1.2.5 (core)",
    "org 2.7.0 (core)",
    "schema 2.3.8 (core)",
    "settings 1.4.4 (core)",
    "sobject 0.1.17 (core)",
    "source 2.10.3 (core)",
    "telemetry 2.1.4 (core)",
    "templates 55.4.9 (core)",
    "trust 2.4.9 (core)",
    "user 2.3.8 (core)"
  ]
}

Additional information

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[junners]

if you also take a look at the documentation, there's something a bit confusing on the flags used there

https://developer.salesforce.com/docs/atlas.en-us.sfdx_cli_reference.meta/sfdx_cli_reference/cli_reference_project_commands_unified.htm#cli_reference_project_deploy_start_unified

the text on the options description states

• RunSpecifiedTests — Runs only the tests that you specify with the --run-tests flag. Code coverage requirements differ from the default coverage requirements when using this test level. Executed tests must comprise a minimum of 75% code coverage for each class and trigger in the deployment package. This coverage is computed for each class and trigger individually and is different than the overall coverage percentage.

but the options for --tests states

-t | --tests TESTS
Optional
Apex tests to run when --test-level is RunSpecifiedTests.

Separate multiple test names with commas, and enclose the entire flag value in double quotes if a test contains a space.

Point being is the other option says --run-tests and the other says --tests

[mshanemc]

I think there might be a few things wrong with these docs.

Can you try this change

instead of --tests "FileUtilitiesTest,GeoCodingServiceTest,TestPropertyController,TestSampleDataController"

use either
--tests FileUtilitiesTest --tests GeoCodingServiceTest --tests TestPropertyController --tests TestSampleDataController
--tests FileUtilitiesTest GeoCodingServiceTest TestPropertyController TestSampleDataController

[git2gus]

This issue has been linked to a new work item: W-13168361

[R0Wi]

@mshanemc can confirm that both workarounds are working for me.

My output of sf version --verbose --json:

{
  "cliVersion": "@salesforce/cli/1.77.1",
  "architecture": "linux-x64",
  "nodeVersion": "node-v18.16.0",
  "osVersion": "Linux 5.10.16.3-microsoft-standard-WSL2",
  "shell": "sh",
  "rootPath": "/usr/bin/sfdx/sf",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 2.1.9 (core)",
    "@oclif/plugin-commands 2.2.14 (core)",
    "@oclif/plugin-help 5.2.9 (core)",
    "@oclif/plugin-not-found 2.3.23 (core)",
    "@oclif/plugin-plugins 2.4.7 (core)",
    "@oclif/plugin-search 0.0.16 (core)",
    "@oclif/plugin-update 3.1.13 (core)",
    "@oclif/plugin-version 1.3.3 (core)",
    "@oclif/plugin-warn-if-update-available 2.0.36 (core)",
    "@oclif/plugin-which 2.2.19 (core)",
    "@salesforce/cli 1.77.1 (core)",
    "apex 2.2.14 (core)",
    "auth 2.7.14 (core)",
    "data 2.3.12 (core)",
    "deploy-retrieve 1.8.18 (core)",
    "info 2.6.8 (core)",
    "limits 2.3.13 (core)",
    "login 1.2.7 (core)",
    "org 2.7.2 (core)",
    "schema 2.3.10 (core)",
    "settings 1.4.5 (core)",
    "sobject 0.1.18 (core)",
    "source 2.10.5 (core)",
    "telemetry 2.1.6 (core)",
    "templates 55.4.11 (core)",
    "trust 2.4.12 (core)",
    "user 2.3.9 (core)"
  ]
}

So for me indeed the docs seem to be wrong here

In my opinion, the deprecation warnings (for example for sfdx force:mdapi:deploy) should also be adjusted to help people migrating to the newer sf CLI.

[junners]

Gotcha, looks like just an issue with docs then, seems like this is true for all array-like parameters as stated in the docs. (reference on --source-dir flag)

Thanks for clarifying, looks like there's nothing really wrong in the cli, it's just that the docs for the --tests flags are a bit misleading
There is an example of an array-like parameter stated on the docs but didn't think it would apply to the --tests.

[mshanemc]

We'll fix help/examples, and also provide a warning. I just had someone else ask about this behavior, and I'd expect it to be a common source of problems and confusion.

[junners]

Seems like the related PR has been merged, closing this issue. Pls do share the release number

[mshanemc]

currently in nightly on both sf (1.79.1) and sfdx (7.202.0)
the highest minor version of those releases will become RC on May 17(sf) | May 18(sfdx),
and promote to latest on May 24(sf) | May 25(sfdx)

[ItaDeutschAvo]

run multiple tests with out comma's.
.... -l RunSpecifiedTests --tests ContentVersionModel_Test FilePreviewController_Test --coverage-formatters json