Failed subtests make test failed - 2

[maxnikulin]

A test with failed subtests reached its finish is not accounted as
passed. Failure and skip counters are incremented by the number
subtests having similar result.

• Collect subtest reports in an Item attribute
• Choose if subtest should be test failure reason in
  pytest_runtest_call and wrap report into a dedicated exception.
• In pytest_runtest_makereport log subtest reports,
  low priority non-successful rest outcome (skip)
  and return subtest failure report unwrapped from the exception.

Drawbacks:

• Distorted order of subtest results. Failure is moved to the end
  even it was followed by some passed or skipped subtests.
• In no capture mode reports are logged after output,
  so it may be hard to guess their relation.

[maxnikulin]

Before: failed test is accounted as passed

============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.4.3, py-1.5.2, pluggy-0.13.1 -- /home/ptst/venv/36-pytest-dev/bin/python
cachedir: .pytest_cache
rootdir: /home/ptst/pytest-subtests-test
plugins: xdist-1.32.0, forked-1.1.3, subtests-0.3.2.dev7+g87b7e03
collecting ... collected 1 item

test_demo.py::test_demo FAILED                                           [100%]
test_demo.py::test_demo PASSED                                           [100%]
test_demo.py::test_demo PASSED                                           [100%]

=================================== FAILURES ===================================
___________________________ test_demo [binary math] ____________________________

subtests = SubTests(ihook=<pluggy.hooks._HookRelay object at 0x7f9bba8b1048>, suspend_capture_ctx=<bound method CaptureManager.gl...state='resumed' _in_suspended=False> _capture_fixture=None>>, request=<SubRequest 'subtests' for <Function test_demo>>)

    def test_demo(subtests):
        with subtests.test("binary math"):
>           assert 1 + 1 == 10
E           assert 2 == 10
E             +2
E             -10

test_demo.py:4: AssertionError
=========================== short test summary info ============================
FAILED test_demo.py::test_demo - assert 2 == 10
========================= 1 failed, 1 passed in 0.03s ==========================

After

============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.4.3, py-1.5.2, pluggy-0.13.1 -- /home/ptst/venv/36-pytest-dev/bin/python
cachedir: .pytest_cache
rootdir: /home/ptst/pytest-subtests-test
plugins: xdist-1.32.0, forked-1.1.3, subtests-0.3.2.dev7+g87b7e03
collecting ... collected 1 item

test_demo.py::test_demo PASSED                                           [100%]
test_demo.py::test_demo FAILED                                           [100%]

=================================== FAILURES ===================================
___________________________ test_demo [binary math] ____________________________

subtests = SubTests(request=<SubRequest 'subtests' for <Function test_demo>>)

    def test_demo(subtests):
        with subtests.test("binary math"):
>           assert 1 + 1 == 10
E           assert 2 == 10
E             +2
E             -10

test_demo.py:4: AssertionError
=========================== short test summary info ============================
FAILED test_demo.py::test_demo - assert 2 == 10
========================= 1 failed, 0 passed in 0.03s ==========================

Test:

def test_demo(subtests):
    with subtests.test("binary math"):
        assert 1 + 1 == 10
    with subtests.test("base >= 2"):
        assert 1 + 1 == 2

See issue #11 and pull request #27

[radkujawa]

what's with with PR? It would be nice to simply use it.

[nicoddemus]

Unfortunately this went under the radar, sorry about that.

I will try to take a look this week.

[nicoddemus]

Closing as we have not reached a consensus yet if this is the way to move forward (See #27).

Thanks again @maxnikulin for the PR!