AnkiDroid unit test suite currently not working on arm64 JDKs on macOS

[mikehardy]

[UPDATE: until solved, we publish rsdroid-arm64.dylib on each release in the list of the artifacts and you can point the robolectric tests to it for use on M1 machines]

---

Symptom:

    java.lang.UnsatisfiedLinkError: /private/var/folders/5b/1q9_lqrs79ng7n9hjm_xl4cm0000gn/T/librsdroid2032698931526253765.dylib: dlopen(/private/var/folders/5b/1q9_lqrs79ng7n9hjm_xl4cm0000gn/T/librsdroid2032698931526253765.dylib, 0x0001): tried: '/private/var/folders/5b/1q9_lqrs79ng7n9hjm_xl4cm0000gn/T/librsdroid2032698931526253765.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/lib/librsdroid2032698931526253765.dylib' (no such file)

@viciousAegis This is a bug on our side. We do not build io.github.david-allison-1:anki-android-backend-testing for M1 Macs.

I have an M1 Mac arriving sometime late March, so I'll be personally invested in testing

As a workaround:

• Remove @RunWith(AndroidJUnit4::class)
• Remove :RobolectricTest()
• This will only work because the test does not depend on the Anki Database

As a workaround, please enable GitHub actions on your GitHub clone of Anki-Android and test there

Properly fixing this

Issue

• We have a library written in rust: librsdroid. This is compiled for Android
• Robolectric doesn't run Android, it runs tests via the JVM on your PC
• Therefore, librsdroid needs to be compiled for native platforms
• We have a separate library: anki-android-backend-testing which contains the dlls and dylibs needed
• When this was written, M1 macs didn't exist

Please see: https://github.com/ankidroid/Anki-Android-Backend/tree/main/rsdroid-testing

• First of all, this isn't well documented. Please submit a PR with better documentation if you want
• Most of the work is performed via GitHub actions
• Feel free to submit a pull request to the GitHub actions get this working.
  • Compile the dylib for arm64e (I have no idea how to do this, but it should be possible)
    • The code should be:

      Anki-Android-Backend/rsdroid-testing/build.gradle

      Lines 184 to 202 in 9302b3f

      	processResources.dependsOn preBuildWindows
      	processResources.dependsOn copyWindowsOutput
      	// To fix: "toolchain 'nightly-x86_64-unknown-linux-gnu' is not installed"
      	// execute in bash: rustup toolchain install nightly-x86_64-unknown-linux-gnu
      	// "linker `x86_64-unknown-linux-gnu-gcc` not found"
      	// sudo ln -s /usr/bin/cc /usr/local/bin/x86_64-unknown-linux-gnu-gcc
      	// rustup target add x86_64-pc-windows-gnu --toolchain nightly
      	// brew install mingw-w64 && x86_64-w64-mingw32-gcc -v
      	processResources.dependsOn preBuildLinux
      	processResources.dependsOn copyLinuxOutput
      	if (org.gradle.internal.os.OperatingSystem.current().isMacOsX()) {
      	// due to restrictions on downloading the MacOS SDK, we can only build for MacOS on MacOS
      	// > Install a reasonable number of copies of the Apple Software on Apple-branded computers
      	// https://www.apple.com/legal/sla/docs/xcode.pdf
      	processResources.dependsOn preBuildMac
      	processResources.dependsOn copyMacOutput
      	}

  • Include two dylibs in the jar
  • Fix this code to select the correct dylib based on your architecture:

    • Anki-Android-Backend/rsdroid-testing/src/main/java/net/ankiweb/rsdroid/testing/RustBackendLoader.java

      Lines 55 to 64 in 9302b3f

      	if (OS.isFamilyWindows()) {
      	load("rsdroid", ".dll");
      	} else if (OS.isFamilyMac()) {
      	load("librsdroid", ".dylib");
      	} else if (OS.isFamilyUnix()) {
      	load("librsdroid", ".so");
      	} else {
      	String osName = System.getProperty("os.name");
      	throw new IllegalStateException(String.format("Could not determine OS Type for: '%s'", osName));
      	}

Originally posted by @david-allison in ankidroid/Anki-Android#10389 (comment)

[david-allison]

Also:

We should be able to cross-compile Rust for arm64e. This /should/ be doable by adding aarch64-apple-darwin in:

Anki-Android-Backend/rsdroid-testing/build.gradle

Lines 109 to 114 in 9302b3f

	case Platform.LINUX: return toWindowsCommandLine("CC=\$ANKIDROID_LINUX_CC ${mainArg} --target x86_64-unknown-linux-gnu")
	case Platform.MACOS:
	/// x86_64-apple-darwin14-cc
	return toWindowsCommandLine("CC=\$ANKIDROID_MACOS_CC ${mainArg} --target x86_64-apple-darwin")
	case Platform.WINDOWS: return toWindowsCommandLine("${mainArg} --target x86_64-pc-windows-gnu")
	}

and installing the target platform: https://github.com/ankidroid/Anki-Android-Backend/blob/9302b3f89e09643e95c42166743f8563a3822ddc/.github/scripts/install_rust_robolectric_targets.sh

And then:

• An option in the .java to detect an M1 and pull the correct dylib
• A fix to to check_robolectric_assets.sh allow 2 dylibs in the .jar

Note that aarch64-apple-darwin is a Tier 2 target in Rust.

ankidroid/Anki-Android#10389 (comment)

[david-allison]

Got this working.

But: cross doesn't support aarch64-apple-darwin yet: cross-rs/cross#508, so we won't be able to build a dylib in CI.

I'm open to suggestions, but for now, I'll build a dylib for arm64 and will include it in the releases folder, along with a PR in Anki-Android to get an override working via environment variable

Before this PR: replace

https://github.com/ankidroid/Anki-Android/blob/f0aa92f54dec9475fe23e18c980fb3fe735bcff1/AnkiDroid/src/test/java/com/ichi2/anki/RobolectricTest.kt#L92

with a call to load the dylib

RustBackendLoader.loadRsdroid("/Users/davidallison/StudioProjects/Anki-Android-Backend/rslib-bridge/target/aarch64-apple-darwin/release/librsdroid.dylib")

EDIT: Added to https://github.com/ankidroid/Anki-Android-Backend/releases/tag/0.1.10

[mikehardy]

osxcross supports arm64e now, so this triple (aarch64-apple-darwin) is probably supportable
Here's how someone in a related project did it: multiarch/crossbuild#51 (comment)

This is the PR for x86_64-apple-darwin support to cross-r/cross: cross-rs/cross#480
I imagine something based on that would build a Docker image that could then be used with cross-rs/cross via an image tag for the triple

[mikehardy]

I want to leave this open for a bit. I just generated the 0.1.11 and 0.1.14-anki2.1.54 artifacts and here was how I did it:

git co 0.1.11 # or your branch name here
git submodule update --recursive
./tools/doctor.sh  # this sets the correct rust version etc, installs targets or should etc
rustup target add aarch64-apple-darwin
cd rslib-bridge
cross build --release --features no-android --verbose --target aarch64-apple-darwin
open target/aarch64-apple-darwin/release/  # now you have a finder window open with rsdroid.dylib there

Open up the tag here on the releases page https://github.com/ankidroid/Anki-Android-Backend/releases
Edit the release
Drop rsdroid.dylib on there and rename it to rsdroid-arm64.dylib

How exactly to integrate it after merging #202 is up in the air - I have not tested - but this should be updated once tested