Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload the Surefire reports as artifacts #16229

Merged
merged 3 commits into from
Apr 4, 2021
Merged

Conversation

gsmet
Copy link
Member

@gsmet gsmet commented Apr 3, 2021

This is part of a pet project I'm working on.

@gsmet gsmet added the area/infra internal and infrastructure related issues label Apr 3, 2021
@quarkus-bot quarkus-bot bot added the area/infra-automation anything related to CI, bots, etc. that are used to automated our infrastructure label Apr 3, 2021
Copy link
Member

@famod famod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe also a retetion period of less than the default 90 days would make sense. 🤷‍♂️

.github/workflows/ci-actions-incremental.yml Outdated Show resolved Hide resolved
if: failure()
with:
name: surefire-reports-native-windows-${{matrix.category}}
path: "**/target/surefire-reports/TEST-*.xml"
Copy link
Member

@famod famod Apr 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
path: "**/target/surefire-reports/TEST-*.xml"
path: "**/target/*-reports/TEST-*.xml"

to also catch failsafe reports. Although not all jobs execute failsafe tests, it wouldn't hurt to use the broader pattern everywhere.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PS: this should work for Gradle:

**/build/test-results/test/TEST-*.xml

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I generalized a bit but not exactly sure how things will go. We will have to experiment with it.

@gsmet gsmet force-pushed the surefire-reports branch from c167a2d to 2738d83 Compare April 4, 2021 10:14
@quarkus-bot
Copy link

quarkus-bot bot commented Apr 4, 2021

Test Failures

⚙️ native-linux-Security3

📦 integration-tests/vault-app

# Tests:    2
+ Success:  0
- Failures: 0
- Errors:   1
! Skipped:  1
io.quarkus.it.vault.VaultInGraalITCase - see on GitHub
java.lang.RuntimeException: java.lang.RuntimeException: Error waiting for test resource future to finish.
	at io.quarkus.test.junit.NativeTestExtension.throwBootFailureException(NativeTestExtension.java:158)
	at io.quarkus.test.junit.NativeTestExtension.beforeEach(NativeTestExtension.java:56)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeEachCallbacks$1(TestMethodTestDescriptor.java:159)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeMethodsOrCallbacksUntilExceptionOccurs$5(TestMethodTestDescriptor.java:195)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:195)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeEachCallbacks(TestMethodTestDescriptor.java:158)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:125)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Caused by: java.lang.RuntimeException: Error waiting for test resource future to finish.
	at io.quarkus.test.common.TestResourceManager.waitForAllFutures(TestResourceManager.java:142)
	at io.quarkus.test.common.TestResourceManager.start(TestResourceManager.java:118)
	at io.quarkus.test.junit.NativeTestExtension.doNativeStart(NativeTestExtension.java:123)
	at io.quarkus.test.junit.NativeTestExtension.ensureStarted(NativeTestExtension.java:92)
	at io.quarkus.test.junit.NativeTestExtension.beforeAll(NativeTestExtension.java:65)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$8(ClassBasedTestDescriptor.java:368)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:368)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:192)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:78)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:136)
	... 34 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Unable to start Quarkus test resource io.quarkus.vault.test.VaultTestLifecycleManager@6807ffe1
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at io.quarkus.test.common.TestResourceManager.waitForAllFutures(TestResourceManager.java:140)
	... 44 more
Caused by: java.lang.RuntimeException: Unable to start Quarkus test resource io.quarkus.vault.test.VaultTestLifecycleManager@6807ffe1
	at io.quarkus.test.common.TestResourceManager.lambda$start$1(TestResourceManager.java:112)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.smallrye.mutiny.TimeoutException
	at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:58)
	at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:61)
	at io.quarkus.vault.runtime.client.VertxVaultClient.exec(VertxVaultClient.java:161)
	at io.quarkus.vault.runtime.client.VertxVaultClient.exec(VertxVaultClient.java:154)
	at io.quarkus.vault.runtime.client.VertxVaultClient.put(VertxVaultClient.java:122)
	at io.quarkus.vault.runtime.client.backend.VaultInternalSystemBackend.init(VaultInternalSystemBackend.java:43)
	at io.quarkus.vault.test.VaultTestExtension.initVault(VaultTestExtension.java:256)
	at io.quarkus.vault.test.VaultTestExtension.start(VaultTestExtension.java:242)
	at io.quarkus.vault.test.VaultTestLifecycleManager.start(VaultTestLifecycleManager.java:33)
	at io.quarkus.test.common.TestResourceManager.lambda$start$1(TestResourceManager.java:107)
	... 5 more

@gsmet
Copy link
Member Author

gsmet commented Apr 4, 2021

Looks like it worked OK. Lucky to have one test failure for this PR :).

@gsmet gsmet merged commit acd3d1e into quarkusio:main Apr 4, 2021
@gsmet
Copy link
Member Author

gsmet commented Apr 4, 2021

Will merge to see how it goes for the other pull requests.

@famod
Copy link
Member

famod commented Apr 4, 2021

Nice! Another gsmetbot. 😁

@gsmet
Copy link
Member Author

gsmet commented Apr 4, 2021

@famod yeah, this is the first feature for which having the bot in Java was really handy as I was able to reuse some existing library namely https://maven.apache.org/surefire/surefire-report-parser/index.html .

@famod
Copy link
Member

famod commented Apr 4, 2021

Really helpful! I think a link to the failed job would make things even more convenient.

@gsmet
Copy link
Member Author

gsmet commented Apr 4, 2021

Yeah, the thing is that it's not so easy and it would require a lot more experiments as things are a bit decoupled in the GH API.

I'll release the code hopefully next week so people will be able to play with it and add new features if they wish.

@gastaldi
Copy link
Contributor

gastaldi commented Apr 4, 2021

@gsmet I think it would be nice if the Tests Failures were set as the first PR comment, for easier access

@gastaldi
Copy link
Contributor

gastaldi commented Apr 4, 2021

How about displaying the tests summary as a table?

Tests: 2

Success Failures Errors Skipped
0 1 0 1

@machi1990
Copy link
Member

@gsmet nice feature.

How about displaying the tests summary as a table?

Tests: 2

Success Failures Errors Skipped
0 1 0 1

+1 @gastaldi I think it will be a nice improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/infra internal and infrastructure related issues area/infra-automation anything related to CI, bots, etc. that are used to automated our infrastructure
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants