verbose output confusing, as it shows test passed #7
Comments
|
This is indeed misleading. If one has 2 subtests there will be report for 3 tests. Another thing is how it will work with test reporting stuff like |
Strange, I get different results here: The e.g, only show the percentage when we reach the end of the test.
I've tried to mimic the output from import unittest
class T(unittest.TestCase):
def test_foo(self):
for i in range(5):
with self.subTest("custom message", i=i):
self.assertEqual(i % 2, 0)
if __name__ == "__main__":
unittest.main()It says above that it executed 1 test, but reported 2 failures. We actually have 5 sub-tests, with 3 of them passing and 2 failures, but the passing sub-tests don't count towards the summary. I followed the same approach to reporting for the Of course we can change the output when running unittest subtests under pytest, or perhaps adopt a different reporting strategy when using |
|
unittest's method of counting things when subTests are involved is just weird. If pytest-subtests's goal is to try to match unittest's behavior, then it's just gonna be weird too. IIUC, accurately bookkeeping the subtests is kind of tricky problem: basically, a test runner really can't(?) know a priori how many subtests there will be in a given test function, because subtests might be generated by iteration, and iterables often don't have a known size until runtime, if even then. Part of the problem also is that the test function itself can actually pass, even if every subtest it contains fails... since execution can reach the end of the function without error, etc. Or, every subtest might pass, but the overall test function could fail, if there are failing asserts in the function that aren't in a |
|
@bskinn that's a good assessment of the problem, thanks. |
|
@nicoddemus Sorry I only have diagnosis to offer, and no concrete ideas for addressing it. :-/ |
|
@bskinn No worries, appreciate the contribution already! 👍 @okken your output example only shows: While I get very different results. Can you confirm that you are indeed using the example from the README? I will try later with Python 3.8 to check if I get a different outcome. |
|
@nicoddemus This is what I get with subtests-0.2.0 |
|
Indeed, I get the same results on Linux. Weird, no idea why there's such a difference between Linux and Windows here, I'll investigate. Thanks! |
|
Found the problem: #10. I will finish the PR later and publish a new release. Thanks everyone! |
|
Sorry for the delay, Sounds like it's understood now. The unittest output is confusing, yes. So my opinion should be obvious by now. With this: import unittest
class T(unittest.TestCase):
def test_foo(self):
for i in range(3):
with self.subTest("custom message", i=i):
self.assertEqual(i % 2, 0)
def test_pass(self):
pass
def test_fail(self):
assert False, 'fail intentionally'c:\sub>python -m unittest -v test_sub.py
test_fail (test_sub.T) ... FAIL
test_foo (test_sub.T) ... test_pass (test_sub.T) ... ok
======================================================================
FAIL: test_fail (test_sub.T)
----------------------------------------------------------------------
Traceback (most recent call last):
File "c:\sub\test_sub.py", line 13, in test_fail
assert False, 'fail intentionally'
AssertionError: fail intentionally
======================================================================
FAIL: test_foo (test_sub.T) [custom message] (i=1)
----------------------------------------------------------------------
Traceback (most recent call last):
File "c:\sub\test_sub.py", line 7, in test_foo
self.assertEqual(i % 2, 0)
AssertionError: 1 != 0
----------------------------------------------------------------------
Ran 3 tests in 0.002s
FAILED (failures=2)with pytest: |
|
Took a look at the xml output, with the --junitxml flag, and it also shows 4 tests But correctly shows 3 |
|
Also, I understand that this is a can-o-worms problem. So, thank you Bruno. I'll take a look at the code and try to understand it. |
|
One more test case that is interesting. import unittest
class T(unittest.TestCase):
def test_fail(self):
with self.subTest():
self.assertEqual(1, 2)No passing subtests, just one failure. Still shows PASSED. |
The example given in the readme is kinda weird when passed the
-vflag.The double
::testis a bit odd.As is the
PASSED.And, well, with 1 test that has 5 subtests, where does the
2 failed 1 passedcome from?Specifically, the
1 passed?The text was updated successfully, but these errors were encountered: