Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[native] Integration test "kafka-snappy" fails with GraalVM 20.3.1.2 #16129

Closed
zakkak opened this issue Mar 30, 2021 · 11 comments · Fixed by #16236
Closed

[native] Integration test "kafka-snappy" fails with GraalVM 20.3.1.2 #16129

zakkak opened this issue Mar 30, 2021 · 11 comments · Fixed by #16236
Assignees
Labels
Milestone

Comments

@zakkak
Copy link
Contributor

zakkak commented Mar 30, 2021

Describe the bug

Native image of integration test kafka-snappy fails with GraalVM 20.3.1.2

Expected behavior

The native image test should pass.

Actual behavior

The native image test fails with:

[INFO] Running io.quarkus.it.kafka.KafkaSnappyProducerITCase
2021-03-31 00:17:44,123 WARN  [org.apa.kaf.cli.adm.AdminClientConfig] (vert.x-worker-thread-0) The configuration 'group.id' was supplied but isn't a known config.
2021-03-31 00:17:44,319 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /kafka failed, error id: ed6debba-ae18-4931-914f-8f37fcc56065-1: org.jboss.resteasy.spi.UnhandledException: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
	at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:106)
	at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:372)
	at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:218)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:519)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
	at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
	at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:138)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.access$000(VertxRequestHandler.java:41)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
	at java.lang.Thread.run(Thread.java:834)
	at org.jboss.threads.JBossThread.run(JBossThread.java:501)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:441)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:302)
	at io.quarkus.it.kafka.KafkaProducerManager.createProducer(KafkaProducerManager.java:28)
	at io.quarkus.it.kafka.KafkaProducerManager.create(KafkaProducerManager.java:39)
	at io.quarkus.it.kafka.KafkaProducerManager_Bean.create(KafkaProducerManager_Bean.zig:347)
	at io.quarkus.it.kafka.KafkaProducerManager_Bean.create(KafkaProducerManager_Bean.zig:363)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:96)
	at io.quarkus.arc.impl.AbstractSharedContext.access$000(AbstractSharedContext.java:14)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:29)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:26)
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:26)
	at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:17)
	at io.quarkus.it.kafka.KafkaProducerManager_ClientProxy.arc$delegate(KafkaProducerManager_ClientProxy.zig:67)
	at io.quarkus.it.kafka.KafkaProducerManager_ClientProxy.send(KafkaProducerManager_ClientProxy.zig:212)
	at io.quarkus.it.kafka.KafkaEndpoint.post(KafkaEndpoint.java:19)
	at java.lang.reflect.Method.invoke(Method.java:566)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
	at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:643)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:507)
	at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:457)
	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:459)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:419)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:393)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:68)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
	... 17 more
Caused by: java.lang.IllegalArgumentException: Unknown  or unsupported compression name: snappy
	at org.apache.kafka.common.record.CompressionType.forName(CompressionType.java:59)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:395)
	... 45 more

To Reproduce

./mvnw clean verify -Dtest-containers -Dstart-containers -Dnative -Dnative.surefire.skip -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-native-image:20.3-java11 -pl integration-tests/kafka-snappy

Environment (please complete the following information):

Output of uname -a or ver

Linux 5.10.20-200.fc33.x86_64

Output of java -version

11.0.10+9

GraalVM version (if different from Java)

20.3.1.2

Quarkus version or git rev

main: 06e7f62

Build tool (ie. output of mvnw --version or gradlew --version)

Maven 3.6.3

@zakkak zakkak added the kind/bug Something isn't working label Mar 30, 2021
@quarkus-bot
Copy link

quarkus-bot bot commented Mar 30, 2021

/cc @cescoffier

@cescoffier
Copy link
Member

Snappy requires graalvm 21.+ because of serialization and method handle support.

@zakkak
Copy link
Contributor Author

zakkak commented Mar 31, 2021

Thanks @cescoffier.

In that case we need a way to detect that the test/app is relying on some unsupported feature and report it to the user. Then the test suite should ideally be able to either skip unsupported tests (based on the GraalVM/Mandrel version) or parse their output and mark them as passing if they fail with the expected message.

@cescoffier
Copy link
Member

that feature waited for Quarkus to switch to GraalVM 21+ as default and prevent using older versions. How did you end up having the issue?

But yes, I agree that we should warn the user in the log, not only on the documentation.

@zakkak
Copy link
Contributor Author

zakkak commented Mar 31, 2021

How did you end up having the issue?

I am using this workflow https://github.com/zakkak/graalvm-quarkus-ci/blob/main/.github/workflows/quarkus.yml to test Quarkus main (and other versions) with various GraalVM versions, so running the workflow with 20.3 picked the test from main's .github/native-tests.json.

@cescoffier
Copy link
Member

cescoffier commented Mar 31, 2021

ah ah.... yeah.... if you remove the safeguard, then :-D

So there are multiple tasks:

  1. better warn the user if they use an unsupported graalvm version (I was thinking we were doing that already, but globally). Maybe we need to have a new build item that advertise the minimal GraalVM version. @geoand @gsmet WDYT?
  2. how can we filter these tests. Can we use Junit with the right predicate DisableIfl?

@geoand
Copy link
Contributor

geoand commented Mar 31, 2021

In JUnit you can have arbitrary predicates.

We could likely just check the GraalVM version and decide whether or not the test should be skipped

@cescoffier
Copy link
Member

Yes, and we have that code somewhere already (in the devtools)

@zakkak
Copy link
Contributor Author

zakkak commented Apr 2, 2021

I gave @DisabledIf a go with no luck.

diff --git a/integration-tests/kafka-snappy/src/test/java/io/quarkus/it/kafka/KafkaSnappyProducerTest.java b/integration-tests/kafka-snappy/src/test/java/io/quarkus/it/kafka/KafkaSnappyProducerTest.java
index b39aac6892..7330285d7e 100644
--- a/integration-tests/kafka-snappy/src/test/java/io/quarkus/it/kafka/KafkaSnappyProducerTest.java
+++ b/integration-tests/kafka-snappy/src/test/java/io/quarkus/it/kafka/KafkaSnappyProducerTest.java
@@ -13,10 +13,12 @@ import org.apache.kafka.common.serialization.IntegerDeserializer;
 import org.apache.kafka.common.serialization.StringDeserializer;
 import org.junit.jupiter.api.Assertions;
 import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIf;
 
 import io.quarkus.test.junit.QuarkusTest;
 import io.restassured.RestAssured;
 
+@DisabledIf(value = "graalVM20OrEarlier", disabledReason = "Kafka Snappy not working in GraalVM < 21.0")
 @QuarkusTest
 public class KafkaSnappyProducerTest {
 
@@ -33,6 +35,10 @@ public class KafkaSnappyProducerTest {
         return consumer;
     }
 
+    public static boolean graalVM20OrEarlier() {
+        return org.graalvm.home.Version.getCurrent().compareTo(21) < 0;
+    }
+
     @Test
     public void test() throws Exception {
         KafkaConsumer<Integer, String> consumer = createConsumer();

The problem seems to be that the JUnit tests are being run using the hosts JVM which results in org.graalvm.home.Version.getCurrent() returning "snapshot". If I change the JAVA_HOME to point to GraalVM then it works.
So we need to:

  1. Detect whether the unit tests are being run against a native image. I didn't find how to do this.
  2. Get the version from the "correct" native-image using: However, to do this we need all the native-image-related properties passed to package to be propagated to the JVM running the JUnit tests.

@geoand
Copy link
Contributor

geoand commented Apr 3, 2021

I'll take a look next week and see what can be done

@geoand
Copy link
Contributor

geoand commented Apr 5, 2021

#16236 takes care of it

@quarkus-bot quarkus-bot bot added this to the 1.14 - main milestone Apr 5, 2021
geoand added a commit that referenced this issue Apr 5, 2021
Disable kafka-snappy tests for versions of GraalVM older than 21.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants