Provide test class orderer to order based on test annotations

[GregJohnStewart]

Description

Currently when testing with different profiles, one needs to be careful about the order of the classes so that you don't have to stand up the Quarkus service more than is necessary.

We should provide a ClassOrderer that developers can enter in junit-platform.properties that can automatically determine the appropriate ordering of test classes based on their annotations.

I would suggest ordering based on all test annotations, not just test profile. Perhaps there could also be a new test class annotation that would let the developer determine their own order.

Implementation ideas

I already do this (for just test profiles) in my project:

@Slf4j
public class TestTypeOrder implements ClassOrderer {
    private static final int ORDER_VAL_NOT_RUNNING = 1;
    private static final int ORDER_VAL_DEFAULT = 2;
    private static final int ORDER_VAL_TEST_PROFILE_EXTERNAL_AUTH = 3;
    private static final int ORDER_VAL_TEST_PROFILE_OTHER = 4;

    @Override
    public void orderClasses(ClassOrdererContext context) {
        log.info("Getting order for classes.");
        context.getClassDescriptors().sort(Comparator.comparingInt(TestTypeOrder::getOrderFor));
    }

    private static int getOrderFor(ClassDescriptor classDescriptor) {
        int order = getOrder(classDescriptor);
        log.info("Order for {}: {}", classDescriptor.getTestClass().getName(), order);
        return order;
    }

    private static int getOrder(ClassDescriptor classDescriptor) {
        if (!classDescriptor.findAnnotation(QuarkusTestResource.class).isPresent()) {
            return ORDER_VAL_NOT_RUNNING;
        }

        Optional<TestProfile> testProfileAnnotation = classDescriptor.findAnnotation(TestProfile.class);
        if (testProfileAnnotation.isPresent()) {
            if (ExternalAuthTestProfile.class.equals(testProfileAnnotation.get().value())) {
                return ORDER_VAL_TEST_PROFILE_EXTERNAL_AUTH;
            }
            return ORDER_VAL_TEST_PROFILE_OTHER;
        }
        return ORDER_VAL_DEFAULT;
    }
}

This will obviously need expanding/ improvements but is along the lines of what I am suggesting to provide.

[geoand]

@famod you have already covered this in QuarkusTestProfileAwareClassOrderer, right?

[famod]

Ah, nice to see that some else has this need! 🙂

Yes, that orderer should cover this request.
It's also mentioned in the docs: https://github.com/quarkusio/quarkus/blob/2.6.0.CR1/docs/src/main/asciidoc/getting-started-testing.adoc#10-testing-different-profiles

Starting from 2.6.0.CR1, this orderer is active by default, which wasn't the case before.

@GregJohnStewart please do check it out and let us know if there's anything you miss. Thanks!

PS: There is at least one more idea for improvement: #21892

[famod]

I'll close this because we already have such an orderer.

Please do file a new issue in case you run into a bug or have ideas for improvement.

[GregJohnStewart]

Ah, did not see this before! Must have not seen that version of the docs when I was looking around somehow. The only other thing I can think of is taking the other test-related annotations into account, but I might leave that to you!

[famod]

other test-related annotations

Which others do you have in mind? As of 2.6.0.CR1 this orderer not only takes @TestProfile in to account but also @QuarkusTestResource (even if in a stereotype).
You can add ordering logic for your custom annotations by overriding this method: https://github.com/quarkusio/quarkus/blob/2.6.0.CR1/test-framework/junit5/src/main/java/io/quarkus/test/junit/util/QuarkusTestProfileAwareClassOrderer.java#L117
(and then register the subclass in your own junit-platform.properties).

[GregJohnStewart]

That actually might be enough for me, I might say potentially the super class of the testclass, but that is because that is how I have implemented things on my end. Profile and testResource is probably fine