The TestSuite value object returned by TestSuite\Filtered::testSuite() contains all tests instead of only the filtered tests

[kennaar]

Q	A
PHPUnit version	10.5.3
PHP version	8.1.9
Installation Method	Composer

Summary

I am implementing a \PHPUnit\Event\TestSuite\FilteredSubscriber. I want to use the list of tests provided by $event->testSuite()->tests() to determine if a test database should be created or not. Currently we are always creating a test database, even when a developer might have provided a filter to run one or multiple unit tests which don't require a real database to be available. This 3 second delay because of the database creation becomes annoying.

Current behavior

$event->testSuite()->tests() is returning ALL tests, without any filtering applied. $event->testSuite()->count() is returning the expected amount.

How to reproduce

Implement an extension using the following code:

<?php

declare(strict_types=1);

namespace Scania\SqsBackend\Tests\Helper\DatabaseManagement;

use PHPUnit\Event\TestSuite\Filtered;
use PHPUnit\Event\TestSuite\FilteredSubscriber;
use PHPUnit\Runner\Extension\Extension;
use PHPUnit\Runner\Extension\Facade;
use PHPUnit\Runner\Extension\ParameterCollection;
use PHPUnit\TextUI\Configuration\Configuration;

final class MyExtension implements Extension
{
    public function bootstrap(Configuration $configuration, Facade $facade, ParameterCollection $parameters): void
    {
        $facade->registerSubscriber(
            new class() implements FilteredSubscriber {
                public function notify(Filtered $event): void
                {
                    // The count matches the amount of tests which will be executed after filtering has been applied
                    $filteredCount = $event->testSuite()->count();
                    // The list of tests is the full list, containing ALL tests defined in phpunit.xml.dist
                    $filteredTestMethods = $event->testSuite()->tests();
                }
            }
        );
    }
}

Expected behavior

I would expect $event->testSuite()->tests() to be returning only the tests which will be executed because of the provided filters.

[sebastianbergmann]

Thank you for your report.

Please provide a minimal, self-contained, reproducing test case that shows the problem you are reporting.

Without such a minimal, self-contained, reproducing test case I will not be able to investigate this issue.

[kennaar]

@sebastianbergmann I have created a minimal project which reproduces the problem. You can find it here: https://github.com/kennaar/phpunit-filtered-event-reproducable

Requirements:
PHP ^8.1
Composer 2.*

Steps to run the project:

git clone [email protected]:kennaar/phpunit-filtered-event-reproducable.git
cd phpunit-filtered-event-reproducable
composer install
composer dump-autoload

# composer scripts
composer test-without-filter
composer test-with-filter
composer test-with-path
composer test-with-group
composer test-with-testsuite
# OR to run all composer scripts in sequence:
composer test-all

The class located at src/FilteredExtension.php is registered as an extension in phpunit.xml. It is registering a \PHPUnit\Event\TestSuite\FilteredSubscriber. This FilteredSubscriber prints debug output the console when the count and actual amount of tests the event object is returning are not equal.

An extra question:

Running phpunit while providing the path to a test or directory containing tests does not trigger the Filtered event. I can kind of see the reasons for this, but it still feels like I am just specifying a filter for phpunit. It's the same case when specifying the --testsuite flag, so the same question for that.

Thanks already for your time!
Looking forward to your answers.

[marac19901990]

I managed to reproduce and fix the issue in #5645.
@sebastianbergmann
@kennaar

[sebastianbergmann]

Closed by #5645.