Introduce QuarkusTestProfileAwareClassOrderer for efficient `@TestProfile` ordering #20156

famod · 2021-09-14T23:22:51Z

Update: Finalized, resolves #20117

@geoand & @stuartwdouglas this is a very early draft for #20117.

I haven't tested it yet but I wanted to get some early feedback:

see TODOs
package; util or somewhere else?
extensibility through the two protected methods

W.r.t. extensibility, one use-case I have in mind for my current project is to also group/sort ArchUnit tests.
Others might want to execute simple unit tests before QuarkusTests (which they could also do by swapping the values of those two junit properties).

Btw, I will add some javadoc and inline comments as well. In case it's hard to see right now: The first QuarkusTestProfile class that is found receives the order number 1, which is added to the base number (1000). The next different QuarkusTestProfile class receives 2 and so on.

PS: I'll probably have to check for @QuarkusIntegrationTest as well.

TODOs:

rethink number based approach for more similarity to continuous testing
cover QuarkusIntegrationTest
add javadoc and extend testing docs

famod · 2021-09-14T23:28:37Z

Btw, I will add some javadoc and inline comments as well.

...and of course I will add something to the testing docs.

stuartwdouglas · 2021-09-15T00:56:40Z

...ork/junit5/src/main/java/io/quarkus/test/junit/util/QuarkusTestProfileAwareClassOrderer.java

+    public void orderClasses(ClassOrdererContext context) {
+        AtomicInteger testProfileCounter = new AtomicInteger();
+        Map<Class<? extends QuarkusTestProfile>, Integer> testProfileNumberMap = new HashMap<>();
+        Function<Class<? extends QuarkusTestProfile>, Integer> testProfileNumberProvider = desc -> testProfileNumberMap


I don't really like the counter approach, as I would prefer something stable between runs.

Ideally this would match the continuous testing sort order (ideally with a shared implementation):

quarkus/core/deployment/src/main/java/io/quarkus/deployment/dev/testing/JunitTestRunner.java

Line 584 in 9950848

List<Class<?>> itClasses = new ArrayList<>();

I think it's stable between runs because e.g. surefire will not change the order randomly unless you tell it to do so.

But I will have a look at what you suggested.

If it is stable then I think the approach is probably fine.

I aligned it a bit more with what you suggested (profile FQCN infix), but I don't really see much value in trying to unify it more since the constraints are different: The orderer has a single list with all tests, the runner in continuous testing has two separate ones. I also wanted to give users the choice to reconfigure the order (or even subclass it).

Btw, I quickly tried this with continuous testing and users might be surprised to see a different ordering there.
But I think we can always follow up with further unifications later.

famod · 2021-09-15T13:58:08Z

FWIW in my current project (with Quarkus 2.2.3 and overridden JUnit) we are now using a slightly simpler version of this orderer and it's working: In one module we save over 20s!

famod · 2021-09-22T15:45:36Z

@geoand I'm almost done revamping this, but let's say I won't make it before Sunday: I suppose this would then be still in time for 2.3.0.Final?

geoand · 2021-09-22T15:46:20Z

To have this in for Final, we would need it in by Tuesday the latest

famod · 2021-09-26T20:31:47Z

...ork/junit5/src/main/java/io/quarkus/test/junit/util/QuarkusTestProfileAwareClassOrderer.java

+
+    protected static final String DEFAULT_ORDER_PREFIX_QUARKUS_TEST = "20_";
+    protected static final String DEFAULT_ORDER_PREFIX_QUARKUS_TEST_WITH_PROFILE = "40_";
+    protected static final String DEFAULT_ORDER_PREFIX_NON_QUARKUS_TEST = "60_";


Note: The "number spacing" might seem odd, but I wanted to leave some gaps for users to fill in their own orderings.

famod · 2021-09-26T20:32:25Z

...ork/junit5/src/main/java/io/quarkus/test/junit/util/QuarkusTestProfileAwareClassOrderer.java

+ * Limitations:
+ * <ul>
+ * <li>This orderer does not (yet) consider {@linkplain io.quarkus.test.common.QuarkusTestResource#restrictToAnnotatedClass()
+ * test resources that are restricted to the annotated class}.</li>


This could be a future improvement.

FTR: #20420

…file` ordering

quarkus-bot · 2021-09-27T00:26:09Z

Failing Jobs - Building `a6883fe`

Status	Name	Step	Failures	Logs	Raw logs
✖	MicroProfile TCKs Tests	`Verify`	Failures	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

⚙️ MicroProfile TCKs Tests #

- Failing: tcks/resteasy-reactive

📦 tcks/resteasy-reactive

✖ Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:exec (test) on project quarkus-tck-resteasy-reactive: Command execution failed.

📦 tcks/resteasy-reactive/target/testsuite/tests

✖ com.sun.ts.tests.jaxrs.platform.container.completioncallback.JAXRSClient0164.argumentIsNullWhenRegistredClassTest line 209 - More details - Source on GitHub

com.sun.ts.tests.jaxrs.common.JAXRSCommonClient$Fault: Unexpected response content No name has been set yet expecting NULL
	at com.sun.ts.tests.jaxrs.platform.container.completioncallback.JAXRSClient0164.assertString(JAXRSClient0164.java:458)
	at com.sun.ts.tests.jaxrs.platform.container.completioncallback.JAXRSClient0164.argumentIsNullWhenRegistredClassTest(JAXRSClient0164.java:209)

famod · 2021-09-27T11:29:50Z

TCKs Tests cannot be related.

@geoand do you want to have a look? I'd like to merge this soon so that it ends up in 2.3.0.Final.

geoand

LGTM, thanks!

stuartwdouglas · 2021-09-27T22:59:19Z

quarkusio/resteasy-reactive-testsuite@a43f921 should fix the intermittent failure.

famod · 2021-09-28T21:26:38Z

quarkusio/resteasy-reactive-testsuite@a43f921 should fix the intermittent failure.

@stuartwdouglas cool, have you updated the ref? See https://github.com/quarkusio/quarkus/blob/main/tcks/resteasy-reactive/README.md?plain=1#L8

famod · 2021-09-28T22:38:49Z

#20437 is taking care of using the latest ref of resteasy-reactive-testsuite

quarkus-bot bot added the area/testing label Sep 14, 2021

stuartwdouglas reviewed Sep 15, 2021

View reviewed changes

quarkus-bot bot added the area/documentation label Sep 26, 2021

famod marked this pull request as ready for review September 26, 2021 20:25

famod changed the title ~~WIP QuarkusTestProfileAwareClassOrderer~~ Introduce QuarkusTestProfileAwareClassOrderer for efficient @TestProfile ordering Sep 26, 2021

famod requested review from stuartwdouglas and geoand September 26, 2021 20:30

famod commented Sep 26, 2021

View reviewed changes

Introduce QuarkusTestProfileAwareClassOrderer for efficient `@TestPro…

a6883fe

…file` ordering

famod added the triage/backport? label Sep 26, 2021

stuartwdouglas approved these changes Sep 26, 2021

View reviewed changes

geoand approved these changes Sep 27, 2021

View reviewed changes

geoand merged commit 9e15c37 into quarkusio:main Sep 27, 2021

quarkus-bot bot added this to the 2.4 - main milestone Sep 27, 2021

famod deleted the test-orderer branch September 27, 2021 12:25

geoand modified the milestones: 2.4 - main, 2.3.0.Final Sep 28, 2021

geoand removed the triage/backport? label Sep 28, 2021

famod mentioned this pull request Sep 28, 2021

Use latest ref of resteasy-reactive-testsuite to stabilize some tests #20437

Merged

radcortez mentioned this pull request Nov 28, 2024

Register a Config instance to bootstrap tests #42715

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce QuarkusTestProfileAwareClassOrderer for efficient `@TestProfile` ordering #20156

Introduce QuarkusTestProfileAwareClassOrderer for efficient `@TestProfile` ordering #20156

famod commented Sep 14, 2021 •

edited

Loading

famod commented Sep 14, 2021 •

edited

Loading

stuartwdouglas Sep 15, 2021

famod Sep 15, 2021

stuartwdouglas Sep 15, 2021

famod Sep 26, 2021 •

edited

Loading

famod Sep 26, 2021

famod commented Sep 15, 2021

famod commented Sep 22, 2021 •

edited

Loading

geoand commented Sep 22, 2021

famod Sep 26, 2021

famod Sep 26, 2021

famod Oct 1, 2021

quarkus-bot bot commented Sep 27, 2021

famod commented Sep 27, 2021

geoand left a comment

stuartwdouglas commented Sep 27, 2021

famod commented Sep 28, 2021

famod commented Sep 28, 2021

Introduce QuarkusTestProfileAwareClassOrderer for efficient @TestProfile ordering #20156

Introduce QuarkusTestProfileAwareClassOrderer for efficient @TestProfile ordering #20156

Conversation

famod commented Sep 14, 2021 • edited Loading

famod commented Sep 14, 2021 • edited Loading

stuartwdouglas Sep 15, 2021

Choose a reason for hiding this comment

famod Sep 15, 2021

Choose a reason for hiding this comment

stuartwdouglas Sep 15, 2021

Choose a reason for hiding this comment

famod Sep 26, 2021 • edited Loading

Choose a reason for hiding this comment

famod Sep 26, 2021

Choose a reason for hiding this comment

famod commented Sep 15, 2021

famod commented Sep 22, 2021 • edited Loading

geoand commented Sep 22, 2021

famod Sep 26, 2021

Choose a reason for hiding this comment

famod Sep 26, 2021

Choose a reason for hiding this comment

famod Oct 1, 2021

Choose a reason for hiding this comment

quarkus-bot bot commented Sep 27, 2021

Failing Jobs - Building a6883fe

Failures

⚙️ MicroProfile TCKs Tests #

📦 tcks/resteasy-reactive

📦 tcks/resteasy-reactive/target/testsuite/tests

famod commented Sep 27, 2021

geoand left a comment

Choose a reason for hiding this comment

stuartwdouglas commented Sep 27, 2021

famod commented Sep 28, 2021

famod commented Sep 28, 2021

Introduce QuarkusTestProfileAwareClassOrderer for efficient `@TestProfile` ordering #20156

Introduce QuarkusTestProfileAwareClassOrderer for efficient `@TestProfile` ordering #20156

famod commented Sep 14, 2021 •

edited

Loading

famod commented Sep 14, 2021 •

edited

Loading

famod Sep 26, 2021 •

edited

Loading

famod commented Sep 22, 2021 •

edited

Loading

Failing Jobs - Building `a6883fe`