[Quarkus main] integration test 'main' fails in native mode: QuarkusTestCallbacksITCase.testCallbackContextIsNotFailed

[jerboaa]

Describe the bug

We see this failure with various Mandrel versions. 23.1, 22.3 and Graal master:

Error:  Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.015 s <<< FAILURE! -- in io.quarkus.it.main.QuarkusTestCallbacksITCase
Error:  io.quarkus.it.main.QuarkusTestCallbacksITCase.testCallbackContextIsNotFailed -- Time elapsed: 0.003 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63)
	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36)
	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31)
	at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:231)
	at io.quarkus.it.main.QuarkusTestCallbacksITCase.testCallbackContextIsNotFailed(QuarkusTestCallbacksITCase.java:61)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

22.3 failure: https://github.com/graalvm/mandrel/actions/runs/7247202176/job/19741555739?pr=638#step:12:651
23.1 failure: https://github.com/graalvm/mandrel/actions/runs/7228893522/job/19699470818#step:12:706
Graal master failure: https://github.com/graalvm/mandrel/actions/runs/7242468774/job/19728465877#step:12:688

[quarkus-bot]

/cc @Karm (mandrel), @galderz (mandrel), @zakkak (mandrel,native-image)

[jerboaa]

Is anybody looking at this? We see considerable CI noise due to it. Why is that flagged housekeeping and area/testing? Is it a test issue?

[zakkak]

I will have a look and try to understand why it's not happening with the published mandrel 23.1 builder image but still shows up in our CI when testing 23.1 source builds.

[zakkak]

The failures at hand are related to either:

a) a failure in another test, e.g. in io.quarkus.it.main.ImageMetricsITCase#verifyImageMetrics
b) a test abort due to a failing assumption, e.g. in

quarkus/test-framework/junit5/src/main/java/io/quarkus/test/junit/nativeimage/NativeBuildOutputExtension.java

Line 54 in 21f055a

Assumptions.assumeFalse(skipVerifyImageMetrics,

This happens because test failures are recorded in the test's extension context through io.quarkus.test.junit.AbstractQuarkusTestWithContextExtension and are then seen by io.quarkus.it.main.QuarkusTestCallbacksITCase#testCallbackContextIsNotFailed which checks to see the test status of the context in

quarkus/integration-tests/main/src/test/java/io/quarkus/it/main/QuarkusTestCallbacksITCase.java

Line 61 in 1ed54f0

assertFalse(TestContextCheckerBeforeEachCallback.CONTEXT.getTestStatus().isTestFailed());

[zakkak]

@geoand why is Quarkus keeping failures from previous tests in the extension context? I guess the callback test is only interested in failures caused in its test class, no?

Furthermore, isn't this way holding only the latest exception? Wouldn't we be interested in all exceptions instead in the case of a failure?

Note: I am considering ignoring failures caused by TestAbortedException in io.quarkus.it.main.QuarkusTestCallbacksITCase#testCallbackContextIsNotFailed (draft PR #38115) but I would prefer resetting the "state" for each IT. WDYT?

[geoand]

I guess the callback test is only interested in failures caused in its test class, no?

exactly