Java: Support Apple Silicon/M1 machines

[jurmous]

When I run an application on an Apple Silicon based JVM depending on the RocksDB JVM library it fails with UnsatisfiedLinkError.

Expected behavior

Maven dependency includes Mac Arm64 compilation

Actual behavior

Maven dependency does not include Mac Arm64 compilation

Steps to reproduce the behavior

Run RocksDB inside a Apple Silicon JVM application. UnsatisfiedLinkError is thrown.

Depends on #7710

Note: RocksDB for Java works normally running in Rosetta mode, but as soon as environment is started in non Rosetta mode the program fails.

[adamretter]

@jurmous I am waiting for #7714 to get merged before I add the update for RocksJava. If you want to test #7714 and give us some feedback that would be great...

[jurmous]

I posted my feedback in #7714.

[jurmous]

@adamretter Thanks for all the work on the Apple Silicon compilation issues! Do you have any update on the RocksJava Apple Silicon support? Or do you need help?

[adamretter]

@jurmous I can likely send a PR tomorrow if that works for you?

[adamretter]

@jurmous okay so there is no problem with building RocksDB and RocksJava from source and running it on Apple Silicon now.

I will shortly send a PR to differentiate between Mac binaries for x64 and aarch64. We can then package both into the next release of RocksJava that is published on Maven central.

As well as separate libraries for each architecture (x64 and aarch64) it seems we could combine them into a single "Universal Binary" (see: https://developer.apple.com/documentation/xcode/building_a_universal_macos_binary), I will give some thought as to the best approach.

[jurmous]

@adamretter Thanks for checking it out!

Cool that there is no problem building the libraries!

Yes both directions would be valid. Universal makes some things easier I guess and the Mac binary is not that large compared to linux64.

[jurmous]

@adamretter Were you already able to give some thought on the best approach? And were you able to create a PR? Thanks! 😊

[adamretter]

@jurmous Sorry for the delay, I do have some preliminary work here for building Universal Binaries. I had to pause whilst other priorities took over, but I hope to get back to it ASAP.

[jurmous]

Ok, I understand. Good luck with the other priorities! Let me know if I can help when you are able to get back to it!

[Yrp]

Hi, guys. Any progress?

[adamretter]

@jurmous @Yrp you can find the start of my work on Universal Binaries for RocksDB here -
https://github.com/adamretter/rocksdb/tree/macos-multi-arch

Running make static_lib_ub will give you a Universal Binary for both
x86_64 arm64.

Building the universal binaries can be done in macOS on either Intel
or Apple Silicon as long as you have macOS bigSur and the latest
XCode.

[jurmous]

@adamretter Thanks for all the work already! ❤️

I have checked out your branch and tried to run make static_lib_ub on root.
I get however an error about an unknown CPU target:
(Big Sur 11.1 & Xcode 12.4 on an M1 machine)

$DEBUG_LEVEL is 1
Makefile:203: Warning: Compiling in debug mode. Don't use the resulting binary in production
  GEN      util/build_version.cc
$DEBUG_LEVEL is 1
Makefile:203: Warning: Compiling in debug mode. Don't use the resulting binary in production
  GEN      util/build_version.cc
TARGET_ARCHITECTURE=x86_64 STATIC_LIBRARY=librocksdb_debug-x86_64.a /Applications/Xcode.app/Contents/Developer/usr/bin/make static_lib
$DEBUG_LEVEL is 0
  GEN      util/build_version.cc
  CC       db/column_family.o
error: unknown target CPU 'armv8-a+crc+crypto'
note: valid target CPU values are: nocona, core2, penryn, bonnell, atom, silvermont, slm, goldmont, goldmont-plus, tremont, nehalem, corei7, westmere,
      sandybridge, corei7-avx, ivybridge, core-avx-i, haswell, core-avx2, broadwell, skylake, skylake-avx512, skx, cascadelake, cooperlake, cannonlake,
      icelake-client, icelake-server, tigerlake, knl, knm, k8, athlon64, athlon-fx, opteron, k8-sse3, athlon64-sse3, opteron-sse3, amdfam10, barcelona, btver1,
      btver2, bdver1, bdver2, bdver3, bdver4, znver1, znver2, x86-64
make[1]: *** [db/column_family.o] Error 1
make: *** [static_lib_ub] Error 2

make static_lib runs fine btw.

This work is for the generic static library if I am correct? If this works correctly further work has to be done to build a universal librocksdbjni-osx.jnilib?

[tobiade]

Hey folks - thanks for the work here. I just ran into this issue when trying to run a Kafka Streams app on my M1 machine. The Kafka Stream library is using RocksDB v5.18.4 under the hood.

@jurmous you mentioned you were able to run RocksDB when running in Rosetta mode? Could you please let me know how you did that? I'm just (very) simply trying to run my app from IntelliJ, so curious as to how to modify it to run in Rosetta mode.

[tobiade]

Actually nvm - I just realised I wasn't running x86 Java.

[jurmous]

@tobiade Ah great that you got it working with Rosetta!
I worked for now with my custom build RocksDB release locally for the M1 so I can run the app natively. I published it back then in jcenter so it will only be there temporary and I only published the Mac target so it cannot be used in a generic fashion on other architectures.
"io.maryk.rocksdb:rocksdbjni:6.17.0"

[plusmancn]

io.maryk.rocksdb:rocksdbjni:6.17.0

@jurmous I found the page shows 403 forbidden on jcenter.
Link: https://jcenter.bintray.com/io/maryk/rocksdb/rocksdbjni/6.17.0

[jurmous]

That page is indeed forbidden but the files should be accessible without being logged in:
https://bintray.com/maryk/maven/rocksdbjni

[adamretter]

@jurmous Sorry for the delay, I had to return the Apple DTK to Apple as they only allow you to have it for a short-time. I have a M1 on order, and so will try and pick it up again once that arrives.

[jurmous]

@adamretter Thanks for the update! As you can read I have for now a temp solution so I can natively do things on my laptop. Thanks for when you can look into it! All your work is greatly appreciated!

[adamretter]

@jurmous I now have an Apple M1, so I will put this back on the TODO list :-)

[pixelsys]

@adamretter thanks, please let us know if a nightly build is available :)

[gmicsa]

Any progress on this? It seems not much left on your side to release M1 compatible version...so looking forward to it :)

[jurmous]

Hey @adamretter,

I have looked into the compilation issue error: unknown target CPU 'armv8-a+crc+crypto' of the universal static library on your branch. I had to dive into how Makefiles work but by trial and error I could nail the issue down. It seems it fails when compiling the x86 static library specifically.

If I comment out the following code in the Makefile the universal static library compiles normally and the lipo command returns both x86_64 and arm64 on the library

ifeq (,$(shell $(CXX) -fsyntax-only -march=armv8-a+crc+crypto -xc /dev/null 2>&1))
CXXFLAGS += -march=armv8-a+crc+crypto
CFLAGS += -march=armv8-a+crc+crypto
ARMCRC_SOURCE=1
endif

So that piece of code needs an extra if check in case of compiling for macOS and X86_64 on an ARM machine.

I guess it only needs an adjustment to wrap this new universal static library inside a jnilib to be bundled for release.

Do you guess you have time for this issue soon? 😄

[martin-g]

@jurmous Has RockDB 6.29 really been released? I saw the release notes three days ago, but rocksdbjni:6.29.3 cannot be downloaded from Maven repository.

@anand1976 I see you created the (Github) release. Do you know why there is no Maven release ?

[anand1976]

@jurmous Has RockDB 6.29 really been released? I saw the release notes three days ago, but rocksdbjni:6.29.3 cannot be downloaded from Maven repository.

@anand1976 I see you created the (Github) release. Do you know why there is no Maven release ?

@adamretter Do you plan to release the Java binary for 6.29.3 soon?

[adamretter]

@LuciferYang @anand1976 Current status is commented on here - #9254 (comment)

[LuciferYang]

Thank you for your reply @adamretter

[jurmous]

RocksDB 6.29.4.1 has been released to Maven Central which should now work on M1 macs.

[adamretter]

Additionally now so has version 7.0.3