quarkus.test.flat-class-path=true required to run Avro tests

[ppalaga]

7ca3307 added quarkus.test.flat-class-path=true to various Avro related test modules see e.g. 7ca3307#diff-6d16e1bd2cec46a1c2d8f00ec4f3476b7554df6379ac769a0492f8aad9f69911R10 Not sure what particular failure symptoms were observed there.

After upgrading to Quarkus 2.0.0.Alpha1, adding quarkus.test.flat-class-path=true is also required in Camel Quarkus avro and avro-rpc tests.

Expected behavior

Avro related tests should pass without quarkus.test.flat-class-path=true

Actual behavior

Here is the exception from Camel Quarkus Avro test:

[INFO] [stdout] 2021-04-26 12:57:14,282 INFO  [io.quarkus] (main) Installed features: [camel-avro, camel-core, camel-direct, camel-support-common, cdi, resteasy, smallrye-context-propagation]
[INFO] [stdout] 2021-04-26 12:57:15,585 ERROR [org.apa.cam.pro.err.DefaultErrorHandler] (executor-thread-1) Failed delivery for (MessageId: 1F330288B6FD514-0000000000000000 on ExchangeId: 1F330288B6FD514-0000000000000000). Exhausted after delivery attempt: 1 caught: java.lang.ClassCastException: class org.apache.camel.quarkus.component.avro.it.Value cannot be cast to class org.apache.avro.generic.IndexedRecord (org.apache.camel.quarkus.component.avro.it.Value is in unnamed module of loader 'app'; org.apache.avro.generic.IndexedRecord is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @12fb8913)
[INFO] [stdout] 
[INFO] [stdout] Message History (complete message history is disabled)
[INFO] [stdout] ---------------------------------------------------------------------------------------------------------------------------------------
[INFO] [stdout] RouteId              ProcessorId          Processor                                                                        Elapsed (ms)
[INFO] [stdout] [route7            ] [route7            ] [from[direct://unmarshalUsingSchemaAvroDsl]                                    ] [         2]
[INFO] [stdout]         ...
[INFO] [stdout] [route7            ] [unmarshal4        ] [unmarshal[org.apache.camel.model.dataformat.AvroDataFormat@69f3651e]          ] [         0]
[INFO] [stdout] 
[INFO] [stdout] Stacktrace
[INFO] [stdout] ---------------------------------------------------------------------------------------------------------------------------------------
[INFO] [stdout] : java.lang.ClassCastException: class org.apache.camel.quarkus.component.avro.it.Value cannot be cast to class org.apache.avro.generic.IndexedRecord (org.apache.camel.quarkus.component.avro.it.Value is in unnamed module of loader 'app'; org.apache.avro.generic.IndexedRecord is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @12fb8913)
[INFO] [stdout]         at org.apache.avro.generic.GenericData.setField(GenericData.java:818)
[INFO] [stdout]         at org.apache.avro.generic.GenericData.setField(GenericData.java:841)
[INFO] [stdout]         at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259)
[INFO] [stdout]         at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:142)
[INFO] [stdout]         at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
[INFO] [stdout]         at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
[INFO] [stdout]         at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
[INFO] [stdout]         at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
[INFO] [stdout]         at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
[INFO] [stdout]         at org.apache.camel.dataformat.avro.AvroDataFormat.unmarshal(AvroDataFormat.java:157)
[INFO] [stdout]         at org.apache.camel.support.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:64)
[INFO] [stdout]         at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:463)
[INFO] [stdout]         at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:179)
[INFO] [stdout]         at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:64)
[INFO] [stdout]         at org.apache.camel.processor.Pipeline.process(Pipeline.java:184)
[INFO] [stdout]         at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:398)
[INFO] [stdout]         at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:96)
[INFO] [stdout]         at org.apache.camel.impl.engine.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:217)
[INFO] [stdout]         at org.apache.camel.impl.engine.SharedCamelInternalProcessor$1.process(SharedCamelInternalProcessor.java:111)
[INFO] [stdout]         at org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:83)
[INFO] [stdout]         at org.apache.camel.impl.engine.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:108)
[INFO] [stdout]         at org.apache.camel.support.cache.DefaultProducerCache.send(DefaultProducerCache.java:190)
[INFO] [stdout]         at org.apache.camel.impl.engine.DefaultProducerTemplate.send(DefaultProducerTemplate.java:176)
[INFO] [stdout]         at org.apache.camel.impl.engine.DefaultProducerTemplate.send(DefaultProducerTemplate.java:172)
[INFO] [stdout]         at org.apache.camel.impl.engine.DefaultProducerTemplate.requestBody(DefaultProducerTemplate.java:413)
[INFO] [stdout]         at org.apache.camel.quarkus.component.avro.it.AvroResource.valueMarshalUnmarshalUsingSchemaAvroDsl(AvroResource.java:80)
[INFO] [stdout]         at org.apache.camel.quarkus.component.avro.it.AvroResource_ClientProxy.valueMarshalUnmarshalUsingSchemaAvroDsl(AvroResource_ClientProxy.zig:298)
[INFO] [stdout]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[INFO] [stdout]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[INFO] [stdout]         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[INFO] [stdout]         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[INFO] [stdout]         at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
[INFO] [stdout]         at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
[INFO] [stdout]         at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:646)
[INFO] [stdout]         at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:510)
[INFO] [stdout]         at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:460)
[INFO] [stdout]         at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
[INFO] [stdout]         at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:462)
[INFO] [stdout]         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:420)
[INFO] [stdout]         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:394)
[INFO] [stdout]         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:69)
[INFO] [stdout]         at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
[INFO] [stdout]         at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
[INFO] [stdout]         at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
[INFO] [stdout]         at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
[INFO] [stdout]         at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
[INFO] [stdout]         at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
[INFO] [stdout]         at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
[INFO] [stdout]         at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:138)
[INFO] [stdout]         at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)
[INFO] [stdout]         at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
[INFO] [stdout]         at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
[INFO] [stdout]         at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
[INFO] [stdout]         at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
[INFO] [stdout]         at java.base/java.lang.Thread.run(Thread.java:834)
[INFO] [stdout]         at org.jboss.threads.JBossThread.run(JBossThread.java:501)

To Reproduce

Reproducing with Camel Quarkus is currently only possible in camel-main branch of Camel Quarkus, which requires a Camel SNAPSHOT. I assume removing quarkus.test.flat-class-path=true from the tests in Quarkus source tree can serve as a reproducer too.

[ppalaga]

I guess this is something for @stuartwdouglas

[ppalaga]

The same kind of issue seems to be happening in Camel Quarkus AtlasMap test. AtlasMap has a compound classloader which allows for passing our preferred class loader, but there is no way to override the ordering of its delegate classloaders. (Actually their compound class loader is using a HashSet for storing the delegates, so the ordering is even not reproducible.) So when a bad ordering happens, is not possible to cast a Person (an application class) instance created by AtlasMap lib from a class loaded via 'Quarkus Base Runtime ClassLoader' to Person class loaded via 'Quarkus Runtime ClassLoader' from the application code.

The compound class loader of AtlasMap (and perhaps also of Avro) apparently assumes that the class loaders do not overlap. The setup in Quarkus JVM test does not meet that assumption. https://quarkus.io/guides/class-loading-reference seems to suggest that it is by design. Could you @stuartwdouglas please confirm that my undestanding is correct?

I wonder how should issues like this and the above be addressed? In case of AtlasMap I can propose changes in how they chain the classloaders. I'll perhaps succeed there, but which other options do I have for projects not ready to accept this kind of changes? The parentFirst option does not look like it could help here.