Undocumented change to ExpectedException in JUnit 4.11 changes default behavior when expected class is AssertionError.

[xrisfg]

Prior to JUnit 4.11, a test like this passes:

@rule
public ExpectedException thrown = ExpectedException.none();

@test
public void test() {
thrown.expect(AssertionError.class);
assertTrue(false);
}

With JUnit 4.11, tests like this suddenly fail with the thrown AssertionError, instead of being intercepted and verified by ExpectedException.

On inspecting the code, we see that the behavior of ExpectedException was changed to conditionally include AssertionErrors as a valid Throwable,and that the default behavior was changed from prior versions to ignore AssertionError. The flag handleAssertionErrors defaults to false; a call to the new handleAssertionErrors() method is needed to force it to true. This call is required even whenAssertionError is explicitly expected in the call to expect().

We have test frameworks that throw AssertionErrors, and unit tests for those frameworks that verify that the expected AssertIionErrors are thrown. All those tests fail with JUnit 4.11, unless the tests are changed to force handleAssertionErrors=true.

1. This change in the default behavior should be documented in the JUnit 4.11 Release Notes. We spent considerable time tracking down why our test framework unit tests were suddenly failing.
2. The behavior should be fixed, either by restoring the previous default behavior (which presumably was changed to fix some other problem?), or to set handleAssertionErrors=true when AssertionError is explicitly given as the expected class.

[marcphilipp]

This change has been introduced in #323.

[marcphilipp]

It seems there was some controversy about the default behaviour for assertion errors:
#323 (comment)

@dsaff @stefanbirkner Do you remember the details?

[xrisfg]

Reading back to #323 and #121, it seems the originally stated problem was that ExpectedException was wrapping thrown Assertion and Assumption violations, so that the original exception was masked. However, in the examples I can find there, in no case was ExpectedException told to expect an AssertionError or AssumptionViolationException.

This comment by @dsaff comes closest to describing the current issue:

#323 (comment)

"It's troubling that this test would fail:

@test public void throwExpectedAssertionError() {
thrown.expect(AssertionError.class);
throw new AssertionError();
}"

The implementation fixes one problem, but introduces another.

[dsaff]

Ugh. Agreed that this is a bug. Sorry.

Could be low-hanging fruit for a contributor with free time. We should block 4.12 on a fix.