Improve internal warning tests #4023
Conversation
jobh
force-pushed
the
check-raising-warnings
branch
from
f92ac92 to
102fcae
Compare
jobh
force-pushed
the
check-raising-warnings
branch
from
102fcae to
c87ed08
Compare
This comment was marked as off-topic.
This comment was marked as off-topic.
jobh
force-pushed
the
check-raising-warnings
branch
from
4b1e688 to
5f8de06
Compare
jobh
force-pushed
the
check-raising-warnings
branch
from
5f8de06 to
f777996
Compare
| except ( | ||
| HypothesisDeprecationWarning, | ||
| HypothesisWarning, |
There was a problem hiding this comment.
Hmm. I wonder if it would be better to just remove this, so that all HypothesisWarnings would be treated as an ordinary test failure in the next clause down?
If you don't see a substantial downside to that, let's try it out - I'm guessing that we originally treated the deprecation warning as a hard failure just to save on the time spent shrinking and never thought it through again.
There was a problem hiding this comment.
The problem is (was) that all we saw in that case was a Flaky with a StopTest cause; no trace of the original HypothesisWarning.
The latest revision fixes this, and then the line can be removed.
Side note: Flaky can be caused by one or two different failures; do you think it would make sense to change it into an ExceptionGroup for better reporting?
There was a problem hiding this comment.
Ooh, yeah, defining it as class Flaky(BaseExceptionGroup, HypothesisException) would be lovely.
There was a problem hiding this comment.
Nice! To clarify, I'm not planning ExceptionGroup work in this PR, but I recorded it in a new issue to remember. This PR is complete, pending review comments and CI.
jobh
force-pushed
the
check-raising-warnings
branch
from
67f79d5 to
16155b5
Compare
jobh
force-pushed
the
check-raising-warnings
branch
from
16155b5 to
7d9b4d4
Compare
jobh
force-pushed
the
check-raising-warnings
branch
2 times, most recently
from
afb4847 to
69de3a3
Compare
jobh
force-pushed
the
check-raising-warnings
branch
from
69de3a3 to
2a0e48d
Compare
Fixes #4021
When a warning is raised (due to
-Werror) during test execution, it may cause the test to raiseFlakyfromStopTest. This makes the error message less informative.This PR makesHypothesisWarningfatal, not onlyHypothesisDeprecationWarning. Because the warnings are meant to be seen.