PHPUnit 10 can we get (back?) the summary of non-successful tests when using testdox

[Ilyes512]

When you use PHPUnit 10 with testdox you don't get a summary anymore at the end. When you got 100's of tests you will need to scroll up and search for the failed test that might be scattered.

I have 2 example outputs of running a test (with failures) in Laravel 9 which uses PHPUnit 9 and Laravel 10 which uses PHPUnit 10:

Laravel 9 with PHPUnit 9:

Laravel 10 with PHPUnit 10:

So of course whenever you only got 2 tests it doesn't matter, but if you got 100's of tests it's allot harder to get an overview of the failed tests.

This might be related to #5304

[sebastianbergmann]

I do not think that this summary of non-successful tests is useful.

[Ilyes512]

To bad because this is clearly something that made the DX worse (when having a production application with 1000's of tests using testdox in CI). 🤷

[MelchiorKokernoot]

Agreed. This summary is much needed

[jmckenna]

Seconded, this summary is still very needed.

[bilogic]

I do not think that this summary of non-successful tests is useful.

@sebastianbergmann Could you elaborate more? Is there a better way to quickly find out which test (among the hundreds) has failed?

[robertmarney]

@sebastianbergmann - For additional context: Some of our test suites have 5000 tests, reading the inline logging in testdox is a major downgrade in experience compared to the previous versions. In the new way we need to review sometimes 15000 lines of results to find the details of the failure :(

[Minhyme]

Quick and dirty for people to add to their CI to somewhat restore functionality:

phpunit --testdox > test ; cat test ; cat test | grep "│\|✘" ; if grep -q "ERRORS\|FAILURES" test; then rm test; exit 1; else rm -f test; fi

[lk77]

that summary was really usefull, good luck finding the failing tests now, i have more than 1000 tests

[sfsaccone]

You can grep failed tests from --testdox-text output (grep '\[ \]' <file>), but that only shows test names, not the files. Is there a log format that puts everything on one line? Otherwise, it must be scripted.

I, too, miss the summary.

I may go back to using PHPUnit Text Explorer in VS Code. It includes an option to show only the failed tests. Running tests via the command line seemed easier than the explorer when --testdox included the summary, but not anymore.

[eric-pfi]

+1 for wanting the summary. When Jenkins runs fail, my default is to look at the end of the console output to see the summary of failed tests; having to search for them is extra unnecessary work.

[lk77]

I have a tip to help find the failing tests :

ctrl+f and search for the ✘ symbol, it's unicode U+2718

on linux, hold left ctrl + shift + u until you have a underlined u and type 2718 then enter, and you should have the ✘ character

[borys-p]

@sebastianbergmann is there a good reason to remove such a useful behavior? If people need to search for unicode characters in the output, then something went terribly wrong.

[tarlepp]

Yeah, that old summary was really useful because you could see those errors easily from that summary - now we need to scroll up to see which test failed and this is really pita when you've lot of test like in my case 5k+

[lk77]

We had no other choice but to revert to phpunit 9

[tarlepp]

We had no other choice but to revert to phpunit 9

Using phpunit 9 is not a real solution - sooner or later you need to upgrade.

The proper solution - imho - would be one of following:

• Bring this feature back to phpunit itself
• Community plugin/extension that add this feature

And - imho - the best solution would be bringing this back to phpunit itself.

I don't recall to see any deprecation notices in this summary within phpunit 9 - @sebastianbergmann should you first deprecate features like this before just removing those?

[sebastianbergmann]

As discussed in #5488, #5518, and #5521: this will be brought back. In fact, this is the first thing I plan to work on in the new year after a couple of days off.

[sebastianbergmann]

Sorry for the noise, my bad. This is not about the TestDox issues tracked in #5488, #5518, and #5521 like I initially understood. What I wrote back in #5318 (comment) still holds: I do not think that summary of non-successful tests is useful.

However, I will consider a PR that brings this summary of non-successful tests back as an opt-in.

[tarlepp]

I do not think that summary of non-successful tests is useful.

So when we have eg. that 5k+ tests and some of those fails, we really need to scroll up all those testdox output to find out which of those tests really failed - it's really bad ux - imho.

[bilogic]

@tarlepp

So when we have eg. that 5k+ tests and some of those fails, we really need to scroll up all those testdox output to find out which of those tests really failed - it's really bad ux - imho.

This example has been raised multiple times, but the answer remains the same without really addressing why.

However, I will consider a PR that brings this summary of non-successful tests back as an opt-in.

I hope you read this line too, because there is a ray of hope now.

[tarlepp]

I hope you read this line too, because there is a ray of hope now.

Yeah, I read that. The point of my comment was just to point out that bad ux with current implementation.

And sure it would be nice to see this feature back (by default or opt-in), but we don't have that yet and as you said there might be some hope for that now.

This example has been raised multiple times, but the answer remains the same without really addressing why.

Yep it would be really nice to have more detailed answer why "it's not useful" for developer point of view that is the most important thing to see - imho

[epdenouden]

@sebastianbergmann wrote

However, I will consider a PR that brings this summary of non-successful tests back as an opt-in.

I will build a proposal for the Testdox summary. Nice way to dive into all the refactoring that has been done.

When working on the refactoring of the colorization internals I run into workflows where I had one unit test fail as expected and then a small zoo of end-to-end tests that use Testdox for validation testing. Happy scrolling:

[test output for 3407 plus suite headers]

FAILURES!
Tests: 3407, Assertions: 8991, Failures: 13, Skipped: 16.

[kocsismate]

Just to give one more example where a summary would be very useful: we have ~13k tests, and it's extremely tedious to find failures where there is only a few of them. We use GitLab, and it truncates PHPUnit output, so we have to rely on the raw output, but that's also difficult parse when colors are enabled.

Originally, we enabled the testdox based output with PHPUnit 9 in order to be able to track fatal errors more easily when the execution of PHPunit is halted immediately, but as far as I could determine it's also a regression in PHPUnit 10 that the testdox output is delayed at the end of the execution so we cannot see which test resulted in the fatal error.

[sebastianbergmann]

we enabled the testdox based output with PHPUnit 9 in order to be able to track fatal errors more easily when the execution of PHPunit is halted immediately

The --debug CLI option is far better suited than TestDox output for tracking fatal errors.

[pooriaazimi]

I agree with the other comments. The summary of errors at the end is by far the most important part of PHPUnit output! We have tens of thousands of unit tests that take a long time to execute. They are run in GitHub Actions after every commit, and we often check back in 5-10 minutes to see which tests failed. In PHPUnit 9, clicking on the "Test Failed" action in GitHub would helpfully take us to the end of the logs with a list of all failing tests. Now we have to spend minutes scrolling through the full list, searching for the ✘ unicode character...

[juergenZeindlinger]

@sebastianbergmann why you don't bring back that summary? As you can see in this thread the people like it and need it. I agree with other comment's that it's really not a good thing to simply remove but not mark functionality as deprecated.

[MazzMazz]

@sebastianbergmann The summary is also very much needed on our side and i can completely agree with @kocsismate . The developer experience decreased a lot after upgrading to PHPUnit10

[sebastianbergmann]

I am sorry about the confusion and frustration I caused with my comments in #5318 (comment) and #5318 (comment).

I never imagined that the TestDox output would be "permanently" configured/enabled in a project. To me, that does make sense, especially in a CI environment. To me, TestDox is (and always was) about documentation, and not about "a different way of showing test results".

I now realize that that many developers actually want to always use the TestDox output and therefore now have a worse user experience than before due to the fact that the TestDox output for non-successful tests is not repeated at the end.

The previous functionality will be brought back as an option (configurable through configuration file and CLI option). The work on this is tracked in #5885.

[bilogic]

Thanks, among all the various PHPUnit outputs, testdox is the most readable (what happened and what went wrong), which I guess is why so many are asking for its return.