Problem running RocksDB and JNI on Mac with M1 chip

[sami-badawi]

flink4s did not work directly on a Mac with a M1 chip. The problem is with RocksDB and JNI.
You can run it in Docker.

I made this change to get it running directly:
src/test/scala/com.ariskk.flink4s/FlinkExecutor.scala

    val useRock = false
    val backend =
      if (useRock)
        new RocksDBStateBackend(s"file:///tmp/flink-${Random.nextString(10)}")
      else
        new HashMapStateBackend()
    env.setStateBackend(backend)

Maybe there could be a way to configure for this?

[ariskk]

Hey @sami-badawi !
This is a JVM issue. This has been fixed in RocksDB's latest version but it is likely that Flink isn't yet using it.
I fixed this issue in the past by installing sdkman and then sdk install java 11.0.13.fx-zulu.
If you make sure 11.0.13.fx-zulu is the default JVM it should work around this issue.

I would be a bit hesitant to NOT use RocksDB in flink4s's testing suite as all production deployments use RocksDB and thus bugs might creep in. Maybe you can contribute a section in flink4s' README detailing your workaround?

[sami-badawi]

I have tried this, but it is not available for Mac with M1:

sdk install java 11.0.13.fx-zulu

Stop! java 11.0.13.fx-zulu is not available. Possible causes:
 * 11.0.13.fx-zulu is an invalid version
 * java binaries are incompatible with your platform

[ariskk]

Hey @sami-badawi
Unfortunately I don't have an M1 anymore to reproduce.
I solved the issue using:

• https://itnext.io/how-to-install-x86-and-arm-jdks-on-the-mac-m1-apple-silicon-using-sdkman-872a5adc050d
• Java: Support Apple Silicon/M1 machines facebook/rocksdb#7720
  The actual problem is not RocksDB or Flink but rather JNI. You need to use x86_64 JDK under Rosetta to be able to run locally (that is until Flink updates its RocksDB bindings).
  SDKman needs some config to find Rosetta SDKs which (I think) isn't on by default.

[MartijnVisser]

I think this problem should now be resolved in Flink 1.17, given that the RocksDB version that Flink uses has been updated to add support for M1s.