Native build fails to use snappy compression

[solsson]

With #3

Exception in thread "pool-4-thread-1" java.lang.IllegalArgumentException: Unknown or unsupported compression type id: 2
	at org.apache.kafka.common.record.CompressionType.forId(CompressionType.java:61)
	at org.apache.kafka.common.record.DefaultRecordBatch.compressionType(DefaultRecordBatch.java:211)
	at org.apache.kafka.common.record.AbstractRecordBatch.isCompressed(AbstractRecordBatch.java:33)
	at org.apache.kafka.common.record.DefaultRecordBatch.isCompressed(DefaultRecordBatch.java:95)
	at org.apache.kafka.common.record.DefaultRecordBatch.streamingIterator(DefaultRecordBatch.java:333)
	at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.nextFetchedRecord(Fetcher.java:1205)
	at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.fetchRecords(Fetcher.java:1242)
	at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.access$1400(Fetcher.java:1096)
	at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:544)
	at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:505)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1256)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1188)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1164)
	at se.yolean.kafka.topicscopy.tasks.CopyByPoll.run(CopyByPoll.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:481)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)

https://github.com/apache/kafka/blob/2.1.1/clients/src/main/java/org/apache/kafka/common/record/CompressionType.java#L175

[solsson]

I fail to reproduce this with 3e861ef and the docker-compose stack, both with the current kafka 2.1.1 image and with solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789.

[solsson]

The failure to reproduce must have been due to kafkacat. A message prioduced using echo "value" | ./bin/kafka-console-producer.sh --broker-list kafka:9092 --topic topic1 --compression-codec snappy triggers the error. Gzip is ok but snappy fails.

[solsson]

oracle/graal#532 has some insight on snappy support but no clear solution. dadae1b gets me:

Error: unsupported features in 4 methods
Detailed message:
Error: Unsupported method org.xerial.snappy.SnappyNative.maxCompressedLength(int) is reachable: Native method. If you intend to use the Java Native Interface (JNI), specify -H:+JNI and see also -H:JNIConfigurationFiles=<path> (use -H:+PrintFlags for details)
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time.
Call path from entry point to org.xerial.snappy.SnappyNative.maxCompressedLength(int): 
	at org.xerial.snappy.SnappyNative.maxCompressedLength(SnappyNative.java)
	at org.xerial.snappy.Snappy.maxCompressedLength(Snappy.java:376)
	at org.xerial.snappy.SnappyOutputStream.hasSufficientOutputBufferFor(SnappyOutputStream.java:267)
	at org.xerial.snappy.SnappyOutputStream.compressInput(SnappyOutputStream.java:378)
	at org.xerial.snappy.SnappyOutputStream.flush(SnappyOutputStream.java:334)
	at org.xerial.snappy.SnappyOutputStream.close(SnappyOutputStream.java:423)
	at org.xnio.IoUtils.safeClose(IoUtils.java:152)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:506)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:481)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
	at com.oracle.svm.core.code.IsolateEnterStub.PosixJavaThreads_pthreadStartRoutine_e1f4a8c0039f8337338252cd8734f63a79b5e3df(generated:0)

Call path from entry point to org.xerial.snappy.SnappyNative.rawCompress(Object, int, int, Object, int):
Call path from entry point to org.xerial.snappy.SnappyNative.rawUncompress(Object, int, int, Object, int): 
Call path from entry point to org.xerial.snappy.SnappyNative.uncompressedLength(Object, int, int): 

[solsson]

Here's why I couldn't find the actual error message in kafka's source: https://github.com/quarkusio/quarkus/blob/0.12.0/extensions/kafka-client/runtime/src/main/java/io/quarkus/kafka/client/runtime/graal/SubstituteSnappy.java

[solsson]

Pushing non-native builds now as workaround for this issue. Current from f012fdc is yolean/kafka-topics-copy:plainjava@sha256:859365abc5efd85f5dbff504ea48126d8a06b301571d084daaa3e42a97bea393.