shadow-cljs fails on M1 Macs due to missing architecture

[jakubgs]

Problem

Running make run-clojure fails on an M1 Mac with:

[email protected]:~/status-mobile % make run-clojure
yarn shadow-cljs watch mobile
Configuring Nix shell for target 'clojure'...
No changes detected.
Done in: 0.510s
yarn run v1.22.19
$ /Users/admin/status-mobile/node_modules/.bin/shadow-cljs watch mobile
shadow-cljs - config: /Users/admin/status-mobile/shadow-cljs.edn
WARNING: The org.clojure/core.async dependency in shadow-cljs.edn was ignored. Default version is used and override is not allowed to ensure compatibility.
The versions provided by shadow-cljs can be found here: https://clojars.org/thheller/shadow-cljs/versions/2.11.4
NPM dependency "react" has installed version "16.13.1"
"17.0.1" was required by jar:file:/Users/admin/.m2/repository/reagent/reagent/1.0.0/reagent-1.0.0.jar!/deps.cljs
NPM dependency "react-dom" has installed version "^16.4.2"
"17.0.1" was required by jar:file:/Users/admin/.m2/repository/reagent/reagent/1.0.0/reagent-1.0.0.jar!/deps.cljs
Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load (ClassLoader.java:-2).
/private/var/folders/__/x311ykg17rqgq2wyl4kn1pdr0001yh/T/jna8753030888504535661.tmp: dlopen(/private/var/folders/__/x311ykg17rqgq2wyl4kn1pdr0001yh/T/jna8753030888504535661.tmp, 0x0001): tried: '/private/var/folders/__/x311ykg17rqgq2wyl4kn1pdr0001yh/T/jna8753030888504535661.tmp' (fat file, but missing compatible architecture (have (unknown,i386,x86_64), need (arm64e)))

Full report at:
/var/folders/__/x311ykg17rqgq2wyl4kn1pdr0001yh/T/clojure-7946575477959985438.edn
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
make: *** [run-clojure] Error 1

Additional Information

This issue was first identified by a dev candidate - @naartjie - on his Mac M1 running MacOS 12.6, and reproduced on our M1 CI hosts running MacOS 12.5.

[jakubgs]

The issue does not appear when simply building the JS bundle with make jsbundle-ios or make jsbundle-android.

So the issue might be specific to either yarn or shadow-cljs:

status-mobile/Makefile

Lines 258 to 260 in 5d775d9

	run-clojure: export TARGET := clojure
	run-clojure: ##@run Watch for and build Clojure changes for mobile
	yarn shadow-cljs watch mobile

[jakubgs]

The architectures of Java and Node binaries are correct:

[email protected]:~/status-mobile % make shell TARGET=clojure
Configuring Nix shell for target 'clojure'...
No changes detected.
Done in: 0.639s

[nix-shell:~/status-mobile]$ lipo -info $(which java)
Non-fat file: /nix/store/wgm3f258naqlsqr6l7nq4paddmn2dglh-zulu8.54.0.21-ca-jdk-8.0.292/bin/java is architecture: arm64

[nix-shell:~/status-mobile]$ lipo -info $(which node)
Non-fat file: /nix/store/z7zh97v0h0h535dzrasq4zc4s4m9v9p8-nodejs-16.17.1/bin/node is architecture: arm64

[jakubgs]

Some possibly related issues and links:

• Error to run sbt on M1 sbt/sbt#6702
• java.lang.UnsatisfiedLinkError on Apple macOS Big Sur (Apple Silicon) Philippus/sbt-dotenv#91
• fat file, but missing compatible architecture have 'i386,x86_64', need 'arm64e' error with bcrypt on Mac M1 Pro chip paramiko/paramiko#1999
• Cannot launch DynamoDB on Mac M1 architecture 99x/serverless-dynamodb-local#277
• https://jira.atlassian.com/browse/CONFSERVER-78633
• https://youtrack.jetbrains.com/issue/SCL-19382
• https://stackoverflow.com/questions/71345783/
• https://stackoverflow.com/questions/72367316/
• https://stackoverflow.com/questions/71237976/
• https://www.josephso.dev/using-jest-dynamodb-in-apple-silicon-platform-workaround/

This is an interesting sentence:

The underlying issue is a JNA (Java Native Access) dependency incompatibility with Apple's M1 chip. Testcontainers transitively depends on this library. Starting with JNA 5.7.0, this issue is resolved.

https://rieckpil.de/java-development-on-an-apple-m1-a-one-year-review/

[naartjie]

Starting with JNA 5.7.0, this issue is resolved.

@jakubgs and we're using JNA 3.2.2, correct?

[jakubgs]

Indeed, you can find it here:

status-mobile/nix/deps/clojure/deps.list

Line 64 in 5d775d9

net/java/dev/jna/jna/3.2.2/jna-3.2.2.jar

But not sure what exactly pulls it in.

[jakubgs]

Seems like it's hawk:

[nix-shell:~/work/status-mobile]$ shadow-cljs --cli-info 2>/dev/null | grep -B2 jna
[hawk "0.2.11"]
  [net.incongru.watchservice/barbary-watchservice "1.0"]
    [net.java.dev.jna/jna "3.2.2"]

[jakubgs]

It appears the hawk dependency comes directly from shadow-cljs, which was actually removed in 2.11.16:
https://github.com/thheller/shadow-cljs/blob/master/CHANGELOG.md#21116---2021-02-03

And we are using 2.10.14:

 > grep '"shadow-cljs"' package.json 
    "shadow-cljs": "^2.10.14"

The commit states:

Only used to be used on macOS since it was slightly faster than the default JVM implementation. However in Big Sur it seems to cause issues and break completely or just be a lot slower.

thheller/shadow-cljs@f3b89b5

[jakubgs]

And indeed upgrading it to 2.11.16 does fix the issue:

...
shadow-cljs - server version: 2.20.5 running at http://localhost:3449
shadow-cljs - nREPL server started on port 7888
shadow-cljs - watching build :mobile
[:mobile] Configuring build.
[:mobile] Compiling ...

But I also had to upgrade OpenJDK from 8 to 11.

And it also removes a bunch of Clojure dependencies:

 % git diff --stat
 nix/deps/clojure/deps.json | 725 +++++++++++++++++++++++++++++++++++-----------------------------------------------------------------------------
 nix/deps/clojure/deps.list | 105 +++++++----------
 nix/overlay.nix            |   2 +-
 package.json               |   2 +-
 yarn.lock                  |  34 ++----

Which is nice.