Embedded Kafka on M1 Mac - UnsatisfiedLinkError

[mjallday]

In what version(s) of Spring for Apache Kafka are you seeing this issue?

For example:

2020.0.2

Describe the bug

Trying to run https://github.com/spring-cloud/spring-cloud-stream-samples/blob/main/kafka-streams-samples/kafka-streams-branching/src/main/java/kafka/streams/branching/KafkaStreamsBranchingSample.java on an M1 mac.

Fails with the following error

2022-08-16 20:49:13,023  INFO kafka-streams-branching-sample-6303aa83-7375-4cb1-958d-4f577cd149a2-StreamThread-1 o.a.k.c.p.KafkaProducer:1189 - [Producer clientId=kafka-streams-branching-sample-6303aa83-7375-4cb1-958d-4f577cd149a2-StreamThread-1-producer]
 Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2022-08-16 20:49:13,024  INFO kafka-streams-branching-sample-6303aa83-7375-4cb1-958d-4f577cd149a2-StreamThread-1 o.a.k.c.c.KafkaConsumer:1074 - [Consumer clientId=kafka-streams-branching-sample-6303aa83-7375-4cb1-958d-4f577cd149a2-StreamThread-1-restore-c
onsumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions
2022-08-16 20:49:13,026  INFO kafka-streams-branching-sample-6303aa83-7375-4cb1-958d-4f577cd149a2-StreamThread-1 o.a.k.s.KafkaStreams:283 - stream-client [kafka-streams-branching-sample-6303aa83-7375-4cb1-958d-4f577cd149a2] State transition from RUNNING to ERROR
2022-08-16 20:49:13,026 ERROR kafka-streams-branching-sample-6303aa83-7375-4cb1-958d-4f577cd149a2-StreamThread-1 o.a.k.s.KafkaStreams:443 - stream-client [kafka-streams-branching-sample-6303aa83-7375-4cb1-958d-4f577cd149a2] All stream threads have died. The instance will be in error state and should be closed.
2022-08-16 20:49:13,027 ERROR kafka-streams-branching-sample-6303aa83-7375-4cb1-958d-4f577cd149a2-StreamThread-1 o.a.z.s.NIOServerCnxnFactory:92 - Thread       StreamsThread threadId: kafka-streams-branching-sample-6303aa83-7375-4cb1-958d-4f577cd149a2-StreamThread-1

TaskManager
        MetadataState:
        Tasks:
 died
java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/9q/jr9c_p057rl8gwbjxs41bcsw0000gn/T/librocksdbjni8212448697688133344.jnilib
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2398)
        at java.base/java.lang.Runtime.load0(Runtime.java:785)
        at java.base/java.lang.System.load(System.java:1979)
        at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
        at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
        at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)
        at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)
        at org.rocksdb.DBOptions.<clinit>(DBOptions.java:21)
        at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:130)
        at org.apache.kafka.streams.state.internals.TimestampedSegment.openDB(TimestampedSegment.java:49)
        at org.apache.kafka.streams.state.internals.TimestampedSegments.getOrCreateSegment(TimestampedSegments.java:50)
        at org.apache.kafka.streams.state.internals.TimestampedSegments.getOrCreateSegment(TimestampedSegments.java:25)
        at org.apache.kafka.streams.state.internals.AbstractSegments.getOrCreateSegmentIfLive(AbstractSegments.java:84)
        at org.apache.kafka.streams.state.internals.AbstractRocksDBSegmentedBytesStore.put(AbstractRocksDBSegmentedBytesStore.java:142)
        at org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(RocksDBWindowStore.java:62)
        at org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(RocksDBWindowStore.java:27)
        at org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:102)
        at org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:32)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.putAndMaybeForward(CachingWindowStore.java:109)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.lambda$initInternal$0(CachingWindowStore.java:89)
        at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:151)
        at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:109)
        at org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:124)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.flush(CachingWindowStore.java:294)
        at org.apache.kafka.streams.state.internals.WrappedStateStore.flush(WrappedStateStore.java:84)
        at org.apache.kafka.streams.state.internals.MeteredWindowStore.lambda$flush$4(MeteredWindowStore.java:200)
        at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:836)
        at org.apache.kafka.streams.state.internals.MeteredWindowStore.flush(MeteredWindowStore.java:200)
        at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:445)
        at org.apache.kafka.streams.processor.internals.StreamTask.prepareCommit(StreamTask.java:357)
        at org.apache.kafka.streams.processor.internals.TaskManager.commit(TaskManager.java:955)
        at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:851)
        at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:714)
        at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)

To Reproduce

mvn clean test

Expected behavior

Test should pass

Sample

https://github.com/spring-cloud/spring-cloud-stream-samples/blob/main/kafka-streams-samples/kafka-streams-branching/src/main/java/kafka/streams/branching/

[garyrussell]

This has nothing to do with Spring or the embedded broker; the problem is with RocksDB used by the stream topology.

at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)
at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)
at org.rocksdb.DBOptions.<clinit>(DBOptions.java:21)
at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:130)

Looks like it's fixed in a newer version. facebook/rocksdb#7720 - 6.29.4.1 perhaps.