Update to 2.1.55 and refactor the build system (#246)

* Move anki submodule to top level

It's not only used to build the bridge, but also to generate TS files,
so top-level makes more sense. It will also fit more nicely into a
cargo workspace.

* Add a missing drop() that recent Rust versions complain about

* Update to 2.1.55 + ninja build

* Refactor the build system

The 2.1.55 desktop release switched away from Bazel to a new build system,
which required some changes to this repo. Making the changes was a bit complicated
due to the complexity of the current build system, and I ended up shaving some yaks
to make things simpler while I was working on the changes.

Changes:

- Building for the current architecture instead of all architectures is now the
default, so getting started is easier, and a build in Android Studio no longer
requires special flags.
- The build-current.sh script has been split into build-aar and build-robo
for the Android and Robolectric parts, and can be used for both single-arch
and multi-arch builds.
- On arm64 Macs, the build scripts now create arm binaries
- In a multi-arch build, both x86 and arm64 Mac libs are built, and they're
merged into a single library. This can be done in CI, so there is no manual step
required for M1 machines anymore.
- The build now uses protobuf and python binaries/libs that the desktop build
downloads, so they don't need to be installed separately.
- The pinned Rust version and Rust targets are automatically installed as
required.
- The per-platform CI builds now build in debug mode and are faster.
- Updated the docs to explain how the NDK can be installed via Android Studio,
instead of via separate command-line tools.
- The cross/docker stuff has been stripped out, as it's of limited use as it
can't target macOS legally. Easier to use GH actions for the multi-arch
builds, and keep things simple for local development.
- Fix lint not being run in CI; caught an API 23 reference.

Bumps #179 (builds on M1 already work, so this may be simpler than expected?)
Bumps #174 (a bunch of the doctor stuff is obsolete; updated HOWTO.md and GH actions should be consulted)
Bumps #27 (I recommend closing this; single-platform is the default for local builds, and
CI runners don't have any extra compute available)

Tentatively closes #109 (didn't see the flake when I was updating the actions)

Closes #235 (translation submodules now automatically synced with anki submodule)
Closes #213 (path based on script now)
Closes #211 (builds for Arm Mac on Arm Macs)
Closes #197 (single arch is the default)
Closes #196 (desktop venv is used)
Closes #195 (can be done via the GUI, and does not require separate cli download)
Closes #168 (latest Rust; easier changing via rust-toolchain.toml)
Closes #164 (universal dylib)
Closes #127 (no docker)
Closes #106 (most of those scripts obsolete; some commands moved into build scripts)
Closes #99 (no docker)
Closes #98 (no docker)
Closes #97 (submodule automatically updated)
Closes #96 (build will fail if commit unavailable)
Closes #53 (no docker)
Closes #40 (DEBUG=1 option documented)
Closes #9 (simpler OOTB experience, and updated docs)

* Tweak print statements in build.gradle

Opted to match the actual variable name rather than make it human
readable; the print statements are just there to help debug issues
when the user forgets to shut down gradle first.

* Elaborate on prerequisites

* Integrate HOWTO inside README; remove more obsolete instructions

* Remove plantuml files

* Add platform guard to multi-arch build

* Fix clippy lints; check in CI

* Include lint in check, and remind users they have to use build-aar first

* Custom path no longer required for rsdroid-testing

* Reduce verbosity of cargo build in aar

* Mention python3 env

* Strip some obsolete stuff from doctor.sh

* Increase timeout to 10 seconds

#246 (comment)

* Mention msys2

* Apply build-all.yml suggestions from Mike

* Increase timeout of multi-arch build

* Tweak cache keys

* Confirm aar/jar output

* Change Mani's PS script to a .bat file

Avoids the need to change the execution policy, and the new script
checks execution success.

* Fix incorrect build target in build-aar.bat

* Use desktop build's bundled Python on Windows

Avoids the need to install it separately

* Add os suffix to artifacts

* Apply suggestions from code review

Co-authored-by: Mike Hardy <[email protected]>

* Only hash lockfiles

https://github.com/ankidroid/Anki-Android-Backend/pull/246/files#r1038175214

* Update to stable release commit

Co-authored-by: Mike Hardy <[email protected]>

.cargo/config.toml

@@ -1,4 +1,7 @@
+[build]
+# override any global customizations, as the gradle plugin requires a specific path
+target-dir = "target"
+
 [env]
-RSLIB_FTL_ROOT = { value = "ftl/core/l10n.toml", relative = true }
-BUILDINFO = { value = "rsdroid/build/generated/anki_artifacts/buildinfo.txt", relative = true }
-BAZEL = "1"
+STRINGS_JSON = { value = "anki/out/strings.json", relative = true }
+PROTOC = { value = "anki/out/extracted/protoc/bin/protoc", relative = true }

.github/workflows/build-all.yml

@@ -0,0 +1,140 @@
+name: Build AAR and Robo (all platforms)
+on:
+  workflow_dispatch:
+    inputs:
+      mavenPublish:
+        description: "Publish artifacts to Maven Central? non-empty to publish, empty to skip publish"
+        required: false
+        type: string
+  push:
+  schedule:
+    - cron: "0 0 * * *"
+  pull_request:
+
+env:
+  ALL_ARCHS: 1
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-release
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: macos-latest
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Fetch submodules
+        run: git submodule update --init
+
+      - name: Setup Ninja
+        uses: ashutoshvarma/setup-ninja@master
+        with:
+          version: 1.10.0
+
+      # The action puts a relative path on the PATH 🙄
+      - name: Make path absolute
+        run: echo $(pwd)/ninja_bin >> $GITHUB_PATH
+
+      - name: Configure JDK 1.11
+        uses: actions/setup-java@v3
+        with:
+          distribution: "adopt"
+          java-version: "11" # minimum for Android API31
+
+      - name: Install Android Command Line Tools
+        uses: android-actions/setup-android@v2
+
+      - name: Install Windows cross compiler
+        run: brew install mingw-w64 && x86_64-w64-mingw32-gcc -v
+
+      - name: Install Linux cross compiler
+        run: |
+          brew tap SergioBenitez/osxct
+          brew install x86_64-unknown-linux-gnu
+          x86_64-unknown-linux-gnu-gcc -v
+
+      - name: Install NDK
+        run: .github/scripts/install_ndk.sh 22.0.7026061
+
+      - name: Rust Cache
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ runner.os }}-rust-release-v5-${{ hashFiles('Cargo.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-rust-release-v5
+            ${{ runner.os }}-rust-release
+
+      - name: Anki cache
+        uses: actions/cache@v3
+        with:
+          path: |
+            anki/out/node_modules
+            anki/out/rust
+            anki/out/extracted
+          key: ${{ runner.os }}-anki-v5-${{ hashFiles('Cargo.lock', 'anki/yarn.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-anki-v5
+            ${{ runner.os }}-anki
+
+      - name: Build Android/All
+        run: ./build-aar.sh
+
+      - name: Build Robolectric/All
+        run: ./build-robo.sh
+
+      - name: Check Compiled Libraries
+        run: >
+          cd rsdroid-testing/assets &&
+          ../../.github/scripts/check_robolectric_assets.sh
+
+      - name: Upload rsdroid AAR as artifact
+        uses: actions/upload-artifact@v2
+        with:
+          name: rsdroid-aar
+          if-no-files-found: error
+          path: rsdroid/build/outputs/aar
+
+      - name: Upload rsdroid-robo JAR as artifact
+        uses: actions/upload-artifact@v2
+        with:
+          name: rsdroid-robo
+          if-no-files-found: error
+          path: rsdroid-testing/build/libs
+
+      # following steps only run on workflow dispatch
+
+      - name: Publish AAR to Maven
+        if: "${{ github.event.inputs.mavenPublish != '' && github.event_name == 'workflow_dispatch'}}"       
+        env:
+          ORG_GRADLE_PROJECT_SIGNING_PRIVATE_KEY: ${{ secrets.SIGNING_PRIVATE_KEY }}
+          ORG_GRADLE_PROJECT_SIGNING_PASSWORD: ${{ secrets.SIGNING_PASSWORD }}
+          SONATYPE_NEXUS_USERNAME: david-allison-1
+          SONATYPE_NEXUS_PASSWORD: ${{ secrets.SONATYPE_NEXUS_PASSWORD }}
+        run: |
+          ./gradlew rsdroid:uploadArchives -DtestBuildType=release -Dorg.gradle.daemon=false -Dorg.gradle.console=plain
+
+      - name: Publish JAR to Maven
+        if: "${{ github.event.inputs.mavenPublish != '' && github.event_name == 'workflow_dispatch'}}"       
+        env:
+          ORG_GRADLE_PROJECT_SIGNING_PRIVATE_KEY: ${{ secrets.SIGNING_PRIVATE_KEY }}
+          ORG_GRADLE_PROJECT_SIGNING_PASSWORD: ${{ secrets.SIGNING_PASSWORD }}
+          SONATYPE_NEXUS_USERNAME: david-allison-1
+          SONATYPE_NEXUS_PASSWORD: ${{ secrets.SONATYPE_NEXUS_PASSWORD }}
+        run: |
+          export ANKIDROID_LINUX_CC=x86_64-unknown-linux-gnu-gcc
+          export ANKIDROID_MACOS_CC=cc
+          export RUST_DEBUG=1
+          export RUST_BACKTRACE=1
+          export RUST_LOG=trace
+          export NO_CROSS=true
+          ./gradlew rsdroid-testing:uploadArchives -Dorg.gradle.project.macCC=$ANKIDROID_MACOS_CC -DtestBuildType=debug -Dorg.gradle.daemon=false -Dorg.gradle.console=plain
+
+      - name: ℹ️Additional Release Instructions (requires human interaction)
+        if: "${{ github.event.inputs.mavenPublish != '' && github.event_name == 'workflow_dispatch'}}"       
+        run: echo "Sign in to https://oss.sonatype.org/#stagingRepositories , close the repository, then release it"

.github/workflows/build.yml

@@ -0,0 +1,126 @@
+name: Build AAR and Robo (single platform)
+on:
+  workflow_dispatch:
+  push:
+  schedule:
+    - cron: "0 0 * * *"
+  pull_request:
+
+env:
+  DEBUG: 1
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+      fail-fast: false
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 80
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Fetch submodules
+        run: git submodule update --init
+
+      - name: Setup Ninja
+        uses: ashutoshvarma/setup-ninja@master
+        with:
+          version: 1.10.0
+
+      # The action puts a relative path on the PATH 🙄
+      - name: Make path absolute
+        run: echo $(pwd)/ninja_bin >> $GITHUB_PATH
+
+      - name: Configure JDK 1.11
+        uses: actions/setup-java@v3
+        with:
+          distribution: "adopt"
+          java-version: "11" # minimum for Android API31
+
+      - name: Install Android Command Line Tools
+        uses: android-actions/setup-android@v2
+
+      - name: Install NDK
+        if: matrix.os != 'windows-latest'
+        run: .github/scripts/install_ndk.sh 22.0.7026061
+
+      - name: Install NDK (Windows)
+        if: matrix.os == 'windows-latest'
+        run: |
+          Write-Host "NDK Install Started"
+          (. sdkmanager.bat --install "ndk;22.0.7026061" --sdk_root="$Env:ANDROID_SDK_ROOT") | out-null
+          Write-Host "NDK Install Completed"
+
+      - name: Rust Cache
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ runner.os }}-rust-debug-v5-${{ hashFiles('Cargo.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-rust-debug-v5
+            ${{ runner.os }}-rust-debug
+
+      - name: Anki cache
+        uses: actions/cache@v3
+        with:
+          path: |
+            anki/out/node_modules
+            anki/out/rust
+            anki/out/extracted
+          key: ${{ runner.os }}-anki-v5-${{ hashFiles('Cargo.lock', 'anki/yarn.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-anki-v5
+            ${{ runner.os }}-anki
+
+      - name: Build Android/X86_64
+        if: matrix.os != 'windows-latest'
+        run: ./build-aar.sh
+
+      - name: Build Android/X86_64 (Win)
+        if: matrix.os == 'windows-latest'
+        run: ./build-aar.bat
+
+      - name: Check Rust (Linux)
+        if: matrix.os == 'ubuntu-latest'
+        run: ./check-rust.sh
+
+      - name: Run tests (Mac)
+        uses: reactivecircus/android-emulator-runner@v2
+        if: matrix.os == 'macos-latest'
+        timeout-minutes: 30
+        with:
+          api-level: 21
+          target: default
+          arch: x86_64
+          profile: Nexus 6
+          script: ./check-droid.sh
+
+      - name: Build Robolectric/X86_64
+        if: matrix.os != 'windows-latest'
+        run: ./build-robo.sh
+
+      - name: Build Robolectric/X86_64 (Win)
+        if: matrix.os == 'windows-latest'
+        run: ./build-robo.bat
+
+      - name: Upload rsdroid AAR as artifact
+        uses: actions/upload-artifact@v2
+        with:
+          name: rsdroid-aar-${{ matrix.os }}
+          if-no-files-found: error
+          path: rsdroid/build/outputs/aar
+
+      - name: Upload rsdroid-robo JAR as artifact
+        uses: actions/upload-artifact@v2
+        with:
+          name: rsdroid-robo-${{ matrix.os }}
+          if-no-files-found: error
+          path: rsdroid-testing/build/libs