Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No warning when CLI options are used multiple times #5704

Closed
ondrejmirtes opened this issue Feb 17, 2024 · 7 comments
Closed

No warning when CLI options are used multiple times #5704

ondrejmirtes opened this issue Feb 17, 2024 · 7 comments
Assignees
Labels
feature/configuration/cli feature/test-runner CLI test runner type/bug Something is broken version/10 Something affects PHPUnit 10 version/11 Something affects PHPUnit 11

Comments

@ondrejmirtes
Copy link

Hi, right now it seems from my testing that passing --filter multiple times rewrites the previous occurrence so the first --filter option is silently discarded.

vendor/bin/phpunit --filter 'PHPStan\\Type\\UnionTypeTest::testSorting' --filter 'PHPStan\\Type\\UnionTypeTest::testAccepts#7' --verbose --debug
PHPUnit 9.5.23 #StandWithUkraine

Runtime:       PHP 8.3.1
Configuration: /Users/ondrej/Development/phpstan/phpunit.xml
Warning:       No code coverage driver available

Test 'PHPStan\Type\UnionTypeTest::testAccepts with data set #7 (PHPStan\Type\UnionType Object (...), PHPStan\Type\ObjectType Object (...), PHPStan\TrinaryLogic Object (...))' started
Test 'PHPStan\Type\UnionTypeTest::testAccepts with data set #7 (PHPStan\Type\UnionType Object (...), PHPStan\Type\ObjectType Object (...), PHPStan\TrinaryLogic Object (...))' ended

Time: 00:00.117, Memory: 570.00 MB

OK (1 test, 1 assertion)

I'd like to use --filter to be interpreted in a way - "I want to run tests that pass this filter and also tests that pass this other filter". So in this case PHPUnit should run the two tests I specified.

Even if you don't want to interpret multiple --filter options that way, I think something could be done about the silent discarding.

Thank you.

@ondrejmirtes ondrejmirtes added the type/enhancement A new idea that should be implemented label Feb 17, 2024
@sebastianbergmann
Copy link
Owner

sebastianbergmann commented Feb 17, 2024

May I kindly ask you to check whether PHPUnit 10.5 and PHPUnit 11.0 also silently ignore multiple --filter options? Thanks!

@sebastianbergmann
Copy link
Owner

Nevermind, I can confirm that --filter (as well as --exclude-filter) can be used only once and that when it is used multiple times the "last one parsed wins", silently ignoring the previously parsed ones. I consider this a bug that should be fixed for PHPUnit 10 and PHPUnit 11 (PHPUnit 9 no longer receives bug fixes).

I do not, however, think that multiple --filter options should be allowed.

@sebastianbergmann sebastianbergmann self-assigned this Feb 17, 2024
@sebastianbergmann sebastianbergmann added type/bug Something is broken version/10 Something affects PHPUnit 10 version/11 Something affects PHPUnit 11 and removed type/enhancement A new idea that should be implemented labels Feb 17, 2024
@sebastianbergmann sebastianbergmann changed the title Passing --filter multiple times No warning when --filter or --exclude-filter options are used multiple times Feb 17, 2024
@ondrejmirtes
Copy link
Author

I do not, however, think that multiple --filter options should be allowed.

Yeah, I got a little bit obsessed recently with passing the same option multiple times after watching https://www.youtube.com/watch?v=aolI_Rz0ZqY (great talk btw) because git blame does different things whether you pass -C once, twice, or thrice. So I thought it could apply here but of course I'm fine if it doesn't do anything useful.

@sebastianbergmann sebastianbergmann changed the title No warning when --filter or --exclude-filter options are used multiple times No warning when CLI options are used multiple times Feb 18, 2024
@sebastianbergmann
Copy link
Owner

sebastianbergmann commented Feb 18, 2024

All long options supported by PHPUnit's CLI test runner can only be used once. For instance, when --filter foo --filter bar is used then only --filter bar is used as it overwrites any --filter option that was given before.

As a first step, PHPUnit's CLI test runner will be changed (for PHPUnit >= 10.5) to emit a warning when an option is used multiple times. Future work will be done (for PHPUnit >= 11.1) to allow multiple --group and --exclude-group options, for example.

@ondrejmirtes
Copy link
Author

Awesome, thanks!

@gnutix
Copy link

gnutix commented Mar 12, 2024

@sebastianbergmann Is does not seem to be true of the -d option, yet I started having a warning about it.
We've used it for years in our test pipeline to pass arguments to our tests. Our commands look like this :

phpunit -c phpunit-something.xml --testsuite=something \
  -d memory_limit=1G \
  -d app.tests.generate=1 \
  -d app.tests.temporal=2024 \
  -d app.tests.xxx=yyy \
  ...

And all arguments have always been properly passed, and still do on PHPUnit 10.5.11 (just tested right now to be sure).

However, we do have (crappy) custom code in our bootstrap.php file to support this, so maybe that's why it still works ?

// https://github.com/sebastianbergmann/phpunit/issues/634
// The upstream issue about passing custom parameters to PHPUnit was closed.
// Let's simply abuse `-d` for now and scan argvs.
/** @var string $arg */
foreach ($_SERVER['argv'] as $arg) {
    if (preg_match('/app\.tests\.generate=1/', $arg)) {
        $_SERVER['TESTS_GENERATE'] = '1';
    }
    if (preg_match('/app\.tests\.temporal=([0-9A-Z:\-\/]+)/', $arg)) {
        $_SERVER['TESTS_TEMPORAL'] = $matches[1];
    }
    // ... and on it goes ...
}

And then we use $_SERVER['TESTS_XXX'] in our test code to trigger different behaviors.

@sebastianbergmann
Copy link
Owner

sebastianbergmann commented Mar 12, 2024

Is does not seem to be true of the -d option, yet I started having a warning about it.

I meant to write "long options" instead of just "options" in #5704 (comment). I totally forgot about -d.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature/configuration/cli feature/test-runner CLI test runner type/bug Something is broken version/10 Something affects PHPUnit 10 version/11 Something affects PHPUnit 11
Projects
None yet
Development

No branches or pull requests

3 participants