Group exclusions are ignored on folder level

[juriansluiman]

In both the phpunit.xml declarations as for the command line arguments, my excluded test are still running. This happens when I try to run a directory of tests, a single file works fine.

So assume I have this phpunit.xml:

<phpunit bootstrap="./Bootstrap.php" colors="true">
    <groups>
        <exclude>
            <group>disable</group>
        </exclude>
    </groups>
</phpunit>

And a testcase which looks like this

/**
 * @group disable
 */
class SomeTest extends \PHPUnit_Framework_TestCase
{    
    public function testMethod()
    {
        $this->assertEquals(true, true);
    }
}

And assume the testcase is loaded fine by the autoloader, then the following occurs:

This works (ie. test does NOT run):

• $ phpunit path/To/SomeTest.php
• $ phpunit -c phpunit.xml path/To/SomeTest.php
• $ phpunit --exclude-group disable path/To/SomeTest.php

This does not work (ie test DOES run):

• $ phpunit path/To/
• $ phpunit -c phpunit.xml path/To/
• $ phpunit --exclude-group disable path/To/

This gave me the following conclusion:

1. Disabling a testcase is working, because:
2. Try to run that specific case by providing the path to the file containing the class works
3. But run (multiple) cases by providing the path to a directory is not working

To complete the info, I use phpunit 3.5.15 which is installed through pear. Though this bug report is a simplified version, this problem occurs with the current zf2 code base (https://github.com/zendframework/zf2). Users with 3.5.14 didn't have this problem, they told me all was working fine.

[edorian]

It seems this has been fixed. My tests for this work fine in 3.6.3 just using phpunit and phpunit tests/otherFolder.

My phpunit.xml looks like the linked one so i think this is solved.

Feel free to reopen if the problem persists for you. Maybe with the complete reproduce code including the folder structure :)

[bootch777]

We faced the same issue. OS: Ubuntu 11.10. PHPUnit installed using apt-get.

$ phpunit --version
PHPUnit 3.6.4 by Sebastian Bergmann.

$ uname -a
Linux shirjaev 2.6.38-13-generic #53-Ubuntu SMP Mon Nov 28 19:33:45 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

[bootch777]

I tried PHPUnit installed by PEAR. The problem is still present.

$ phpunit --version
PHPUnit 3.6.10 by Sebastian Bergmann.

[edorian]

Like stated without a reproduce case there is nothing much I can do

[diogobaeder]

We were having the same "problem" here, so we created a repository to try to reproduce the "bug": https://github.com/diogobaeder/phpunit-bug-358

But, actually, /eriksencosta discovered that the tests excluded by "--exclude-group" are not being run, which is the expected behaviour, but the number of tests being shown at each pointed line end is not consistent with the total number of tests which is printed at the execution.

For example, this is the output of the complete test suite of our internal project:

PHPUnit 3.6.10 by Sebastian Bergmann.

............................................................. 61 / 1505 ( 4%)
....................................S.S...................... 122 / 1505 ( 8%)
............................................................. 183 / 1505 ( 12%)
.......................................................I..... 244 / 1505 ( 16%)
............................................................. 305 / 1505 ( 20%)
............................................................. 366 / 1505 ( 24%)
............................................................. 427 / 1505 ( 28%)
............................................................. 488 / 1505 ( 32%)
............................................................. 549 / 1505 ( 36%)
............................................................. 610 / 1505 ( 40%)
.............SS.SS....S...................................... 671 / 1505 ( 44%)
............................................................. 732 / 1505 ( 48%)
............................................................. 793 / 1505 ( 52%)
............................................................. 854 / 1505 ( 56%)
............................................................. 915 / 1505 ( 60%)
............................................................. 976 / 1505 ( 64%)
............................................................. 1037 / 1505 ( 68%)
............................................................. 1098 / 1505 ( 72%)
............................................................. 1159 / 1505 ( 77%)
............................................................. 1220 / 1505 ( 81%)
............................................................. 1281 / 1505 ( 85%)
................................................I............ 1342 / 1505 ( 89%)
............................................................. 1403 / 1505 ( 93%)
.....S....................................................... 1464 / 1505 ( 97%)
........................

Time: 03:16, Memory: 503.75Mb

OK, but incomplete or skipped tests!
Tests: 1488, Assertions: 3444, Incomplete: 2, Skipped: 8.

Therefore I think this is not properly a "bug", but a bad communication about the number of tests, which can be improved if we show the correct number ("1488" instead of "1505", at the case above)

[juriansluiman]

I am not sure what it was, because I did not conclude this purely on statistics. In the Zend Framework test suite some tests grouped under @group disable were throwing exceptions, which remained when I excluded this "disable" group.

However, with above repository I forked I cannot reproduce this anymore. Meanwhile I am on phpunit 3.6.10 so this can stay closed, perhaps it was a bug but solved between 3.5.15 and 3.6.10 :)

[edorian]

The test count numbers can be off. That happens because with the current test runner not all the filters (like exclude group) are applied before the tests are started but they should not be executed.

If it works for you with 3.6.10 thats great :)

Thanks for the reproduce @diogobaeder.