Discrepancy in statistics (number of tests) when some tests are excluded.

[bantu]

When running PHPUnit from https://github.com/phpbb/phpbb3 the default configuration in phpunit.xml.dist disables some slow tests. It however seems like the number of disabled tests is included in the progress lines.

PHPUnit 3.5.14 by Sebastian Bergmann.

...............................................................  63 / 910 (  6%)
............................................................... 126 / 910 ( 13%)
............................................................... 189 / 910 ( 20%)
............................................................... 252 / 910 ( 27%)
............................................................... 315 / 910 ( 34%)
............................................................... 378 / 910 ( 41%)
............................................................... 441 / 910 ( 48%)
............................................................... 504 / 910 ( 55%)
............................................................... 567 / 910 ( 62%)
............................................................... 630 / 910 ( 69%)
............................................................... 693 / 910 ( 76%)
............................................................... 756 / 910 ( 83%)
............................................................... 819 / 910 ( 90%)
............................................................... 882 / 910 ( 96%)
..............

Time: 22 seconds, Memory: 20.50Mb

OK (896 tests, 10210 assertions)

Note that 910 should be 896 tests.

[edorian]

This is still easily reproducible with 3.6.2 and master

phpunit
PHPUnit 3.6.2 by Sebastian Bergmann.

Configuration read from /home/edo/Desktop/PHP/phpunit-dev/phpbb3/phpunit.xml.dist

I............................................................   61 / 1278 (  4%)
.............................................................  122 / 1278 (  9%)
.............................................................  183 / 1278 ( 14%)
.............................................................  244 / 1278 ( 19%)
.............................................................  305 / 1278 ( 23%)
.............................................................  366 / 1278 ( 28%)
.............................................................  427 / 1278 ( 33%)
.............................................................  488 / 1278 ( 38%)
.............................................................  549 / 1278 ( 42%)
.............................................................  610 / 1278 ( 47%)
.............................................................  671 / 1278 ( 52%)
.............................................................  732 / 1278 ( 57%)
.............................................................  793 / 1278 ( 62%)
.............................................................  854 / 1278 ( 66%)
.............................................................  915 / 1278 ( 71%)
.............................................................  976 / 1278 ( 76%)
............................................................. 1037 / 1278 ( 81%)
............................................................. 1098 / 1278 ( 85%)
............................................................. 1159 / 1278 ( 90%)
............................................................. 1220 / 1278 ( 95%)
........................................

Time: 16 seconds, Memory: 30.50Mb

OK, but incomplete or skipped tests!
Tests: 1260, Assertions: 10807, Incomplete: 1.

[edorian]

From what I've tried it is not worth hacking this into the current implementation of the TestRunner. The numbers are collected before the test are run but the exclude group filtering happens during execution. Applying the filtering before the execution (or for the counting) would work but seems to be a lot of effort to get right for this issue.

This should solve its self once #10 is implemented.

This allows the number of tests that are to be run to be calculated correctly before the tests are actually run

So I'm closing this as a "will be solved as part of another issue"