JBake fails to run or build on Apple Silicon

[mhberger]

Hi there

Thank you for JBake – I have been using it since June 2015 (that is when
I get the urge to write something pithy at Code is Mandatory)

Recently I started using a MacBook Air (with Apple M1 Silicon) for
processing and unfortunately JBake does not run or build on Apple M1
Silicon. It works fine on Apple Intel.

Exceptions and references to the underlying issue are listed below.

Digging in it is appears to be an issue with OrientDB.

As a result of nosing around and to solve this itch, I have started work
on removing the dependency of OrientDB and using JDBC and Groovy Sql with
the implementation done in SQLite.

The work is ongoing on a “Prototype PR” type branch that is really rough as guts at the moment.

Exceptions caused trying to run

$ jbake -b
JBake v2.7.0-rc.2 (2021-05-24 22:30:41[GMT+01:00] 045c61c#) [http://jbake.org]

14:02:31.341 INFO  org.jbake.app.Oven - Baking has started...
Warning: Nashorn engine is planned to be removed from a future JDK release
14:02:31.485 INFO  c.o.common.jna.ONative - Default limit of open files (512) will be used.
14:02:31.518 INFO  c.o.common.jna.ONative - 17179869184 B/16384 MB/16 GB of physical memory were detected on machine
14:02:31.518 INFO  c.o.common.jna.ONative - Detected memory limit for current process is 17179869184 B/16384 MB/16 GB
14:02:31.519 INFO  c.o.o.c.e.OMemoryAndLocalPaginatedEnginesInitializer - JVM can use maximum 4096MB of heap memory
14:02:31.519 INFO  c.o.o.c.e.OMemoryAndLocalPaginatedEnginesInitializer - Because OrientDB is running outside a container 12% of memory will be left unallocated according to the setting 'memory.leftToOS' not taking into account heap memory
14:02:31.519 INFO  com.orientechnologies - OrientDB auto-config DISKCACHE=10,321MB (heap=4,096MB os=16,384MB)
14:02:31.519 INFO  c.o.o.c.e.l.OEngineLocalPaginated - System is started under an effective user : `mark`
14:02:31.520 INFO  c.o.o.c.e.l.OEngineLocalPaginated - Allocation of 156879 pages.
14:02:31.529 ERROR jbake - An unexpected error occurred: /private/var/folders/43/bt5j2vf520v6n9gnx968cs840000gn/T/jna-3344077/jna5212213165608407388.tmp: dlopen(/private/var/folders/43/bt5j2vf520v6n9gnx968cs840000gn/T/jna-3344077/jna5212213165608407388.tmp, 1): no suitable image found.  Did find:
	/private/var/folders/43/bt5j2vf520v6n9gnx968cs840000gn/T/jna-3344077/jna5212213165608407388.tmp: no matching architecture in universal wrapper
	/private/var/folders/43/bt5j2vf520v6n9gnx968cs840000gn/T/jna-3344077/jna5212213165608407388.tmp: no matching architecture in universal wrapper
mhbmacxiv jbake (master) $

Exceptions caused trying to run tests

GroovyTemplateEngineRenderingTest > renderTags FAILED
    org.gradle.internal.exceptions.DefaultMultiCauseException: Multiple Failures (3 failures)
    	java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
    	java.lang.NullPointerException: Can't set default locale to NULL

Related Git Hub Issues

Test failures on Apple silicon (aarch64) #1323
Add darwin-aarch64 support #1297
Enable building jna for Darwin arm64 #1238
Support for Apple silicon #20
Fix variadic arguments on arm64 darwin ABI #577
Enable Strict Type Enforcement for Dynamic Method Dispatching

Software environment

# uname -a
Darwin mhbmacxiv 20.5.0 Darwin Kernel Version 20.5.0: Sat May  8 05:10:31 PDT 2021; root:xnu-7195.121.3~9/RELEASE_ARM64_T8101 arm64

# sw_vers
ProductName:	macOS
ProductVersion:	11.4
BuildVersion:	20F71

# java --full-version
openjdk 11.0.11+9-LTS


# jbake version (running lastest snapshot)
JBake v2.7.0-rc.2 (2021-05-24 22:30:41[GMT+01:00] 045c61c#) [http://jbake.org]

# jbake version (trying to build)
$ git log | head
commit 109b7c1379b907172acfee7815d945d3c1c5c51b
Merge: 045c61c 5829541
Author: Jonathan Bullock <[email protected]>
Date:   Tue Jun 8 22:48:33 2021 +0100

    Merge pull request #707 from jonbullock/fix/692-template-folders

    Fix to allow templates to be stored in folders

[jbaron]

Facing same issue, is there already some idea/agreement if moving to sqlite for future releases might be the right approach?

@mhberger saw that you continue to update your fork. Is that fork more or less usable on M1 ?

[mhberger]

Hi. In reply to about whether it is more or less usable, it builds (using
SQLite) on M1, Intel, Linux and Windows.

I use it mainly on M1. Once I unzip the installaiton and set path, it
works building my blog and the JBake website.

# Build
./gradlew clean smoketest assemble

# The distributions are in jbake-dist/build/distributions. Unzip jbake-2.7.0-rc.4-bin.zip and add
# jbake-2.7.0-rc.4-bin/bin to path and then you should be able to use jbake
#

But to get it polished and merged back into master/main will require some
tidying up and and deciding on what to do for the future.

Currently the work on the Prototype/PR branch includes the following

• Uses JDBC and SQLite insted of OrientDB
• Upgraded to use Gradle 7
• Upgraded to use JDK 11
• Upgraded to use JUnit 5
• Got it working with TeamCity Cloud Beta. But then once the beta
  finiished, had to stop – the monthly ongoing cost is a bit steep.
• Got it (sort of) working with Azure Pipelines, but have not pursued
  this.

[jbaron]

Thanks, just tried it out and compiles and runs fine on my M1 and even feels faster (btw my website uses asciidoc + freemarker)

But it seems it doesn't handle custom doc types yet. But this could also be a 2.7.x compatibility issue since previously I used the maven plugin and I guess that is still on the 2.6.x version. Need to invest a bit more.

[mhberger]

K. Have you got an example using custom doc types? It could be that I missed something moving away from OrientDB.

[jbaron]

I guess the easiest way would be to clone my website (small one):

git clone https://github.com/neurallayer/roboquant.org.git
cd roboquant.org
jbake . docs

But if a dedicated example is more useful, just let me know and I'll make one.

[mhberger]

That is great. Thanks. Let me take a look and I will see if I can figure it out.

[jbaron]

P.S the default type is a custom type. So these two lines in the jbake.properties files are relevant:

template.doc.file=doc.ftl
default.type=doc

[mhberger]

Thanks again. I understand what is supposed to happen. And I can see in the code where it is “failing”.

I will find the functionality and associated test (if there is not one will try and put something together).

[jbaron]

That's is quick, thanks!!! If there are some test or samples that required that I can help with, just let me know.

[mhberger]

Doh! I missed a call to updateDocTypesFromConfiguration(). This has been fixed and a test added. I have just used the templates and a page from your website as the test resource. Let me know if this does not suit. Or if we can simplify.

[jbaron]

Just tested it also and it works great!!!

My website generates correctly and also very fast. Possible part of this is due to the M1 chip, but I also think SQLite might have less overhead, especially for smaller sites.

[jbaron]

@jonbullock Would you consider moving (or supporting) JBake to SQLLite in the future, or are there obstacles in doing so?

[mhberger]

Hi.

I have created a new branch (mhb/sqlite-2022-01-04) off the old one
(mhb/sqlite-2021-05-30) and then applied new changes from master.

The use case I have been focusing on is unzipping the dist zip and
running that. i.e. I have not tested/used the plugins, mainly because I
don’t use maven day to day. Any pointers to how to test/use the plugins
more than welcome.

Have tested it on Apple M1 and Apple Intel with my blog and with
jbake.org and seems to be working okay.

I need to think of a way of tidying up the branch to make it easier to
understand the changes. Something like this article

Updated: fixed typo.

[jonbullock]

Apologies for the delay in replying to this.

First off let me say thanks to you both for spending time looking into this. I'm not adverse to supporting a different content store other than OrientDB (maybe via a config option so end users have a choice?). However, a non-relational data store (Document support in OrientDB) was selected to enable users to have flexibility in the schema when it comes to content and it's metadata. I've not looked at the SQLite implementation yet but this is the major consideration that came to mind. If SQLite didn't support a flexible schema this would be a hurdle as it would negatively impact users.

[jbaron]

I also didn't look into the details of the SQL Lite implementation, but I'm having some custom content types and properties and they seem to be working fine. A simple key/value pair table in traditional RDBMS can go a long way to support flexible schemes ;)

P.S For me one of the biggest benefit besides it is now also running on Apple M1, is the speed improvements.

[jonbullock]

Very true, could be solved in that way.

From the brief research I've done the root cause of the issue seems to be in JNA that OrientDB uses, but I can't find any reported issue in the OrientDB project. I'll raise one to see what the project maintainers say about it. I hadn't realised a fork of OrientDB had occurred recently too.

How much improvement are you seeing with SQLite?

[jbaron]

I didn’t time the improvements, but if I have some time will give it a go. But from just watching the console output it was clearly noticeable.

[mhberger]

Hi Jon. Thanks for feedback.

re SQLite/JDBC. Although it is a relational, it is being used as a
content store similar to the OrientDB approach, with custom types being
stored in a JSON blob. Part of the work was to create a ContentStore
interface and use that.

The ContentStore interface should allow changing implementations.
Have also used JDBC and Groovy SQL and DataSource to provide a bit more
independence. It should be relatively painless to switch to another
implementation.

At runtime, which contentstore to use is controlled via jbake.properties
by setting jbake_db_implementation=SQLite or jbake_db_implementation=OrientDB.
The default on the branch is SQLite.

The DocumentModel is still the main interface between parsers/templates
etc and the ContentStore. It is translated to a Document which maps to
the underlying DB table structure.

re speed. This requires care when comparing. One has to take account the
speed improvement due to Apple Silicon. The following gives some idea
building a simple blog and jbake.org.

Site	Hardware	Database	Bake Info
simple blog	iMac Pro Intel	OrientDB	Baked 47 items in 4044ms
simple blog	iMac Pro Intel	SQLite	Baked 47 items in 2839ms
simple blog	MacBook Pro M1 Max	SQLite	Baked 47 items in 1836ms
jbake.org	iMac Pro Intel	OrientDB	Baked 196 items in 10731ms
jbake.org	iMac Pro Intel	SQLIte	Baked 196 items in 15461ms
jbake.org	MacBook Pro M1 Max	SQLite	Baked 196 items in 9611ms

I am going to work on tidying up the commits probably organising them
into something along the following lines:

• introduce db.package
• introduce a ContenStore interface
• add sqlite as a database engine
• remove deprecated methods
• convert junit tests to use junit5
• add integration tests

[lvca]

Hi guys, OrientDB is not supported anymore. We are working on ArcadeDB, a fork of OrientDB with an active community and a new engine that is much lighter and faster than OrientDB maintaining the same SQL and similar API. Also, ArcadeDB supports Cypher and GraphQL in case you need it, and much more.

Last but not least, it's embeddable in Java, exactly like OrientDB and it has zero or minimal dependencies based on the configuration used.

If you're interested I'd be happy to help in the update to ArcadeDB, just give me some hint where the repository layer is located and I can do a quick POC ;-)

[jonbullock]

Hi @mhberger

Thanks for the stats regarding speed.

It's more the custom metadata per content file than custom types of content I'm concerned about: https://jbake.org/docs/2.6.7/#custom_metadata

[jonbullock]

Hi @lvca

The big question is does ArcadeDB work on Apple Silicon?

[lvca]

Hi @jonbullock yes it does. It's 100% pure Java, there is no JNI, JNA, or other esoteric configuration like OrientDB ;-)

[lvca]

I've replaced OrientDB with ArcadeDB in the PR #745. It was easy and quick, the SQL is the same, just a little change in the API.

[lprimak]

Same issue here, a blocker because these M1 macs are exploding in popularity

[lprimak]

I did manage a workaround by patching JNA in jbake with the latest JNA distribution

[kwin]

@jonbullock Is this supposed to be fixed with the upcoming 2.7.0?

[lprimak]

Simplest way to resolve this is to just upgrade the versions of JNA in the next JBake release

[kwin]

@lprimak Thanks for the hint, can you raise a PR?

[lprimak]

@kwin probably not in a timely fashion

[kwin]

OrientDB migrated from JNA to JNR in orientechnologies/orientdb@74276df and was recently updated to a version supporting Apple Silicon (orientechnologies/orientdb#9762). The most recent release 3.1.16 should contain that fix (https://github.com/orientechnologies/orientdb/commits/3.1.x)

[yeroc]

Is there anything that can be done to move this forward? I was just trying to setup a new website using JBake and ran into this roadblock so now I'm looking for another Java-based alternative supporting Asciidoc unless I can somehow get this issue unblocked somehow...

[lprimak]

@yeroc #709 (comment)

[yeroc]

@lprimak Can you elaborate on how you patched it? Were you able to force the JNA version in the gradle config? Or did you update the jar file manually somehow? FWIW, I'm trying to use the Maven plugin.

[lprimak]

I just put latest JNA JARs into binary distribution

[yeroc]

@lprimak Ugh ok, not so easy to do that when using via a Maven plugin. Is there any prospect of a proper fix landing?

[FDelporte]

I'm dealing with same issue while evaluating different static site generators...

Seems the version update of OrientDB by @kwin #709 (comment) should resolve the problem? But it isn't part of the next release yet, correct or did I overlook it in the release notes?

[shalk]

I meet the same issue when jbake -b
11:43:04.519 ERROR jbake - An unexpected error occurred: Can't load library: /var/folders/64/smwps7jx2h9092cxzq1t2n3r0000gn/T/jna-109399851/jna13279260892339242457.tmp

Maybe ArcadeDB is good choice for apple sillicon user.

[jbaron]

For those that are using the Maven plugin, you can add the following to the jbake-maven-plugin dependencies section in your pom and then it works (at least on my M1). This will force the plugin to use a newer version of OrientDB library that works on M1.

	<dependency>
		<groupId>com.orientechnologies</groupId>
		<artifactId>orientdb-core</artifactId>
		<version>3.1.16</version>
	</dependency>

Normally this dependencies section already exists and has your template engine(s) libraries mentioned, so just add this to that list.

Later versions (3.2.x) of the lib don't work since JBake use the default username/password for OrientDB which will throw an error (and only a warning in the 3.1.x versions).

[raducotescu]

@jbaron, I think @kwin had some issues with JBake when he tried to update OrientDB, so it might not be recommended to do this - see #755.

[jbaron]

Had indeed the same experience (also just added a comment to #755) , but wondering if that is perhaps junit specific?OrientDB doesn't want to create the DB with the provided name in those failed test case, but perhaps the junit tmp directory is not yet ready (would be strange) or too long or wrong permissions ????

[raducotescu]

No idea, but I think #745 is more promising.

For now I just went around this issue by using https://multipass.run/ and running the JBake build inside an Ubuntu VM. The same could probably be achieved with Docker.

[lprimak]

I can put up my build with latest JNA on my nexus server if there is any interest

[jbaron]

@lprimak I assume that would be the JBake library not the maven plugin correct? (since I believe those are resolved against the Apache Maven repo)

[lprimak]

Not sure. I’d have to look. It I don’t see why it wouldn’t work.

[lprimak]

Ok, I got maven plugin to work as well by using <dependencies> section of the plugin to upgrade JNA to the latest version

Complete project that works on Apple Silicon available here:
https://github.com/flowlogix/jbake-maven-plugin-example

[FDelporte]

This issue is closed, but I still seem to hit the same error. Also with the pom-example shared above by @lprimak

ERROR jbake - An unexpected error occurred: Can't load library: /var/folders/np/6j1kls013kn2gpg_k6tz2lkr0000gn/T/jna-26589803/jna4216443807947299074.tmp

[fvanwesterop]

Same here, on an Apple Macbook Pro M1 , using JBake 2.6.5 or newer:

java.lang.UnsatisfiedLinkError: (blabla JNA blabla...) : tried: '/private/var/folders/8q/lqlsfbh934v75wff63tmhky00000gn/T/jna-97692050/jna8664779561218046705.tmp' (fat file, but missing compatible architecture (have (i386,x86_64), need (arm64e))).

...or some similar error msg, depending on JBake version used. Bottomline is that OrientDB wants some JNA lib but only an Intel x86_64 version is found, while (ofcourse) an ARM64 version is needed on a Mac M1 (or M2).

However: When using 2.6.0 ... 2.6.4 it's working fine WITH OrientDB!
(This was easy to test using sdkman to switch jbake versions :-) )

[jonbullock]

Try the latest release candidate available here: https://github.com/jbake-org/jbake/releases

It includes a later version of OrientDB which supports Apple Silicon.

[lealceldeiro]

This issue is closed, but I still seem to hit the same error. Also with the pom-example shared above by @lprimak

ERROR jbake - An unexpected error occurred: Can't load library: /var/folders/np/6j1kls013kn2gpg_k6tz2lkr0000gn/T/jna-26589803/jna4216443807947299074.tmp

Indeed, this is still an issue. I'm facing this problem on M2

[lprimak]

Can you try jbake-examples-plugin again? I enabled dependabot and updated a whole bunch of dependencies. I have been running consistently on M1 mac and it works perfectly.

[lealceldeiro]

Can you try jbake-examples-plugin again? I enabled dependabot and updated a whole bunch of dependencies. I have been running consistently on M1 mac and it works perfectly.

After updating jbake dependency to 2.7.0-rc.7 it worked properly. That's good news!
The only thing is it makes me a bit nervous being on a rc, and not a "stable" version of jbake.

Thanks @lprimak