Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix skipTest inside subTest #169

Merged
merged 31 commits into from
Dec 7, 2024
Merged
Changes from 10 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
c62d3ae
fix
ydshieh Nov 18, 2024
c5c02fa
fix more: non subtest skip cases
ydshieh Nov 18, 2024
a439226
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 18, 2024
866c990
fix: # type: ignore[method-assign]
ydshieh Nov 18, 2024
590624d
Merge branch 'fix_skip' of https://github.com/ydshieh/pytest-subtests…
ydshieh Nov 18, 2024
a737260
fix more: # type: ignore[attr-defined]
ydshieh Nov 18, 2024
2e38614
fix more: # type: ignore[attr-defined]
ydshieh Nov 18, 2024
d7f0fa5
fix more: # type: ignore[attr-defined]
ydshieh Nov 18, 2024
aa4a17b
fix more: # type: ignore[attr-defined]
ydshieh Nov 18, 2024
d6e9e5e
fix more: # type: ignore[attr-defined]
ydshieh Nov 18, 2024
a99ea3b
fix None exec_info in subtest error
ydshieh Nov 18, 2024
07ef703
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 18, 2024
67993fe
remove copy
ydshieh Nov 20, 2024
23929b5
style
ydshieh Nov 20, 2024
7461dd6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 20, 2024
92e427c
comment
ydshieh Nov 20, 2024
a9c852e
fix python versions
ydshieh Nov 21, 2024
f9c0fba
fix recursion
ydshieh Nov 21, 2024
aecfe7a
fix recursion
ydshieh Nov 21, 2024
87fcb67
Add first test
ydshieh Nov 21, 2024
01c2b12
Add first test
ydshieh Nov 21, 2024
450e81e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 21, 2024
fd94cb9
add more tests
ydshieh Nov 21, 2024
f75438d
add more tests
ydshieh Nov 21, 2024
d14628b
Merge branch 'fix_skip' of https://github.com/ydshieh/pytest-subtests…
ydshieh Nov 21, 2024
15612c0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 21, 2024
9c4af2d
Fix tests due to column size
nicoddemus Dec 7, 2024
3b6c4ff
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 7, 2024
35711d8
Update CHANGELOG
nicoddemus Dec 7, 2024
475a356
Fix tests due to column size
nicoddemus Dec 7, 2024
da27259
Import protected _SubTest locally
nicoddemus Dec 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions src/pytest_subtests/plugin.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

import copy
import sys
import time
from contextlib import contextmanager
Expand All @@ -13,6 +14,7 @@
from typing import Mapping
from typing import TYPE_CHECKING
from unittest import TestCase
from unittest.case import _SubTest # type: ignore[attr-defined]

import attr
import pluggy
Expand Down Expand Up @@ -98,6 +100,24 @@ def _from_test_report(cls, test_report: TestReport) -> SubTestReport:
return super()._from_json(test_report._to_json())


def _addSkip(self: TestCaseFunction, testcase: TestCase, reason: str) -> None:
if isinstance(testcase, _SubTest):
self._originaladdSkip(testcase, reason) # type: ignore[attr-defined]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems we don't really need call self._originaladdSkip for subtest skips, i.e. the following also works

        try:
            raise pytest.skip.Exception(reason, _use_item_location=True)
        except skip.Exception:
            exc_info = sys.exc_info()
            self.addSubTest(testcase.test_case, testcase, exc_info)

if self._excinfo is not None:
exc_info = self._excinfo[-1]
self.addSubTest(testcase.test_case, testcase, exc_info) # type: ignore[attr-defined]
else:
# The non-subtest skips have to be added by `_originaladdSkip` only after all subtest failures are processed by
# `_addSubTest`.
if (
len(
[x for x, y in self.instance._outcome.errors if isinstance(x, _SubTest)]
)
== 0
):
self._originaladdSkip(testcase, reason) # type: ignore[attr-defined]
ydshieh marked this conversation as resolved.
Show resolved Hide resolved


def _addSubTest(
self: TestCaseFunction,
test_case: Any,
Expand All @@ -122,10 +142,27 @@ def _addSubTest(
node=self, call=call_info, report=sub_report
)

# Add non-subtest skips once all subtest failures are processed by # `_addSubTest`.
non_subtest_skip = [
(x, y)
for x, y in self.instance._outcome.skipped
if not isinstance(x, _SubTest)
]
subtest_errors = [
(x, y) for x, y in self.instance._outcome.errors if isinstance(x, _SubTest)
]
if len(subtest_errors) > 0 and len(non_subtest_skip) > 0:
ydshieh marked this conversation as resolved.
Show resolved Hide resolved
last_subset_error = subtest_errors[-1]
if exc_info is last_subset_error[-1]:
for testcase, reason in non_subtest_skip:
self._originaladdSkip(testcase, reason) # type: ignore[attr-defined]
ydshieh marked this conversation as resolved.
Show resolved Hide resolved


def pytest_configure(config: pytest.Config) -> None:
TestCaseFunction.addSubTest = _addSubTest # type: ignore[attr-defined]
TestCaseFunction.failfast = False # type: ignore[attr-defined]
TestCaseFunction._originaladdSkip = copy.copy(TestCaseFunction.addSkip) # type: ignore[attr-defined]
ydshieh marked this conversation as resolved.
Show resolved Hide resolved
TestCaseFunction.addSkip = _addSkip # type: ignore[method-assign]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _addSkip in unittest.case doesn't contain logic to call addSubTest for outcome.skipped (unlike _feedErrorsToResult), and it turns out this causes the reporting issue described in the PR.

Unlike addSubTest (where original TestCaseFunction doesn't have it), addSkip is defined TestCaseFunction, and here we save the original implementation and patch TestCaseFunction.addSkip to a newly defined _addSkip that allows us to call addSubTest.

The _originaladdSkip allows us to restore to it in pytest_unconfigure as well as it turns out necessary to call it too inside the newly defined _addSkip.


# Hack (#86): the terminal does not know about the "subtests"
# status, so it will by default turn the output to yellow.
Expand Down Expand Up @@ -154,6 +191,9 @@ def pytest_unconfigure() -> None:
del TestCaseFunction.addSubTest
if hasattr(TestCaseFunction, "failfast"):
del TestCaseFunction.failfast
if hasattr(TestCaseFunction, "_originaladdSkip"):
TestCaseFunction.addSkip = copy.copy(TestCaseFunction._originaladdSkip) # type: ignore[method-assign]
ydshieh marked this conversation as resolved.
Show resolved Hide resolved
del TestCaseFunction._originaladdSkip


@pytest.fixture
Expand Down