Rich test runner status prompt #528
Replies: 7 comments
-
For now: just print "x test passed, y test failed" at the end of the test. |
Beta Was this translation helpful? Give feedback.
-
Cross-referencing #561 |
Beta Was this translation helpful? Give feedback.
-
In my opinion it's a bit too much, especially for now. I don't see need for this and it looks complicated. What I need is: if test passed or failed, and (only if failed) why it failed (can be exception as is or just an instruction that couldn't be completed). A list of tests with status and reason of failure would be enough for now. |
Beta Was this translation helpful? Give feedback.
-
Cross-referencing #602. |
Beta Was this translation helpful? Give feedback.
-
Cross-referencing flutter/flutter#115874. |
Beta Was this translation helpful? Give feedback.
-
Live-streaming of native tests lifecycle changes (started, failed, finished) could be implemented with:
Output I have in mind (test name format taken from #1619 (comment)):
This would be only a development time feature. Patrol CLI would connect to the running app using the existing Dart VM service connection (see #593 and #630 for an example). |
Beta Was this translation helpful? Give feedback.
-
I know you, @jBorkowska, don't think much of this idea, but I must say it kinda fixes all my current issues with patrol :) It's not always an option for me to use the adb logcat - especially not if the test has been run on a CI/CD environment - but if I could backtrace the events that patrol triggers on my app it would fix everything. |
Beta Was this translation helpful? Give feedback.
-
Intro
Tools like Maestro and Fluttium print test status in the form of easy to comprehend, animated test runner prompt. Go to their websites to see more.
I think Patrol's users would benefit from having a similar task runner.
This is not a priority right now, but I want to start some discussion on this topic to get some feedback.
Idea
The sample Patrol test
would result in this test runner prompt:
t=0 (start)
t = 1 (somewhere in the middle)
t=2 (end)
The above is a pretty blunt copy from Maestro, but I don't think there's lot to be done better.
Implementation
For now, I just want to shed some light on a single problem:
Finder.toString()
prints what the finder found, not what it represents. See an example below:Let's see what
Finder.toString()
prints:Problems with it:
The verbose output is OK for widget tests that developers write, but it is of
no use to testers.
The
toString()
actually evaluates the finder and prints what it found in thewidget tree, instead of printing its representation.
In other words, to have nice output, we need a way to differentiate between
finder and finder result.
I don't like how Flutter doesn't make this distinction. I made two examples to
demonstrate what I mean:
Simple finder example
Current output:
Output I'd like:
Nested finder example
Current output:
Output I'd expect:
😕
We could create some extension method
Finder.repr()
which would return the"Output I'd like" in the above examples, but it's not as simple as that. The
Finder
class has many subclasses, some of them beingprivate.
Caveats
We will only be able to show the past actions and the current one. We don't know what happens in the future. Maestro and Fluttium know that because they interpret their test files. We just execute code.
This might overlap with our early plans to create a task runner GUI (either as a separate desktop Flutter app, similar to
convenient_test_manager
, or as a VS Code extension).Animated task runner prompt could cause problems in environments where the output is not a tty (see Disable animations in the terminal #498 and this to better understand what I mean). We will
Beta Was this translation helpful? Give feedback.
All reactions