RocksDB fails to build on M1 Apple Silicon due to incorrect ifdefs

[hdevalence]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Expected behavior

RocksDB builds on M1 Apple Silicon (in this case via rust-rocksdb bindings)

Actual behavior

RocksDB does not build, erroring as follows:

  running: "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.8.1/fused-src/" "-I" "snappy/" "-I" "lz4/lib/" "-I" "zstd/lib/" "-I" "zstd/lib/dictBuilder/" "-I" "zlib/" "-I" "bzip2/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-DSNAPPY=1" "-DLZ4=1" "-DZSTD=1" "-DZLIB=1" "-DBZIP2=1" "-DNDEBUG=1" "-DOS_MACOSX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/Users/hdevalence/code/zcash/zebra/target/debug/build/librocksdb-sys-10baa88e9458bc13/out/rocksdb/options/options_parser.o" "-c" "rocksdb/options/options_parser.cc"
  cargo:warning=rocksdb/util/crc32c.cc:502:18: error: use of undeclared identifier 'isSSE42'
  cargo:warning=  has_fast_crc = isSSE42();
  cargo:warning=                 ^
  cargo:warning=rocksdb/util/crc32c.cc:1232:7: error: use of undeclared identifier 'isSSE42'
  cargo:warning=  if (isSSE42()) {
  cargo:warning=      ^
  cargo:warning=rocksdb/util/crc32c.cc:1233:9: error: use of undeclared identifier 'isPCLMULQDQ'
  cargo:warning=    if (isPCLMULQDQ()) {
  cargo:warning=        ^
  cargo:warning=3 errors generated.
  exit code: 1

A few lines up is this ifdef (pointed out by @teor2345:

rocksdb/util/crc32c.cc

Line 407 in 29f7bbe

// Detect if ARM64 CRC or not.

It looks like these functions are meant to always be defined, and return false when the intrinsics don't exist, but instead they are missing. Or, perhaps there is some other variable that should be defined on this platform that isn't being set?

Steps to reproduce the behavior

Presumably this is more widely applicable, but here are the specific steps I took:

• Use a Mac with an M1 chip
• Build cmake from source using its bootstrap
• Build LLVM from the bootstrapped cmake (to avoid an x86 cmake running under rosetta2 from misdetecting the host target)
• Set clang-sys environment variables to point to LLVM https://github.com/KyleMayes/clang-sys#environment-variables
• Build Rust code using the rocksdb crate as a dependency

[hdevalence]

Hmm, it looks like here

rocksdb/util/crc32c.cc

Line 497 in 29f7bbe

#elif defined(__linux__) && defined(HAVE_ARM64_CRC)

, there's a path for ARM64_CRC, but only on Linux. Is there some Linux-specific behavior intended?

[hdevalence]

This patch seems to resolve the issue, I'm not sure why ARM64 support is linux-conditional: hdevalence@ed1882e

More context in this comment: rust-rocksdb/rust-rocksdb#482 (comment)

[adamretter]

@hdevalence Hi Henry I have been seeing the same issue on the Apple DTK (A12Z). I started looking into it but didn't spend much time on it yet.

I'm not sure why ARM64 support is linux-conditional

I suspect that was simply because the ARM64 platforms that were available for testing/dev were Linux.

This patch seems to resolve the issue

I will apply the patch and run some tests on ARM Mac and ARM Linux and report back...

[adamretter]

@hdevalence For the DTK I needed a few more changes than you suggested.

Would you be willing to try the attached patch please?
arm64_crc.patch.txt

[adamretter]

Closed by #7714