Building Crystal for ARM architectures

[xiy]

I previously mentioned this in #crystal-lang, but I'd like to investigate using Crystal for scripting in my embedded adventures. I'm currently using an ARM-based prototyping board but the versions of Crystal available are only built for x86/64 architectures.

Can anybody give me any pointers on where to start with building Crystal for ARM?

Thanks!

[asterite]

Hi @xiy,

It would be really nice to have Crystal running on ARM :-)

There's some explanation on how to do cross-compilation here.

We haven't use it for a while, so if you have troubles with it let us know. And, of course, we will definitely help you along the way.

For cross-compiling I would start with very basic programs (Hello World, mandelbrot) and then try to compile more complex code until you get to cross-compile the compiler.

There's also a small program that successfully ran on Windows so you can try that too. Just note that that file's comments are a bit outdated: add "--prelude=empty" to the command explained in the docs and you'll be fine.

A question: what ARM system you have? The other day we had an Arduino in our hands but it only had 2KB memory, and considering that Crystal has a GC we don't think it would be very feasible to run Crystal on it. But maybe you ARM-based prototyping board has more memory.

[xiy]

Hi @asterite,

So far I've attempted to build hello_world.cr with the following command:

crystal build samples/hello_world.cr --cross-compile "Linux armv7l" --target "arm-unknown-linux-gnueabihf"

It will only output an X86-64 object at the moment though, no matter what I give it. If I specify an --mcpu "cortex-a8" it'll also complain about that CPU not matching the target (even though I know it does!) and continue to produce an X86-64 object:

'cortex-a8' is not a recognized processor for this target (ignoring processor)

I have a feeling it's one of two things:

1. My host LLVM isn't set up properly and the Crystal compiler is using a default of X86 targets.
2. This line (https://github.com/manastech/crystal/blob/e165c509b062d32745e0aea80ce221c30db7d13a/src/compiler/crystal/codegen/target_machine.cr#L7) will need to add a call for more than just X86 targets, as in ruby-llvm.

I'm currently suspecting (2) as it rejecting the CPU for that target tells me there's something happening around there.

As for the board, I'm using a pcDuino (Cortex-A8/Allwinner A10-based) that has a whopping 1GB RAM 😄 - It's pretty quick and can handle a bashing. I've managed to natively compile node.js on it, hence my eagerness to get Crystal up and running on it.

[asterite]

Wow, that was quick!

I also suspect (2) is the problem. I remember I tried to cross compile to a 32-bits architecture and also faced some problems, but since we are also using x86_64 here we didn't look at it much longer.

I'll try to investigate further, though I'm sure @waj will have a better answer (but we'll probably have to wait some days due to this time of the year 😸)

[xiy]

A bit of progress over the holidays, but I'm now onto a new problem. I've rebuilt the Crystal compiler with my instructions added to initialize ARM targets. It compiles nicely, and produces an ARM object:

2048.o: ELF 32-bit MSB relocatable, ARM, version 1 (GNU/Linux), not stripped

However you'll notice it's building it as Big Endian (MSB). On link, it complains that the binary was compiled for Big Endian systems when it requires Little Endian, which I found a bit strange. Even in the tech specs for the A10 it states that it's happy with both types! 😕

Anyhoo, I've taken that opportunity to build a toolchain that can be used to compile things properly for ARM using crosstool-ng - it should also make it easier to cross-compile over to X86/Windows at some point - so far so good, but I'm back at work now so things might slow down a little!

[asterite]

Great progress!

One more thing you can try is to generate an .ll file instead of a .o file, then copy that to your ARM board, install LLVM there and using llc and others LLVM binaries generate the .o file. I guess that would generate a correct .o file.

To do that, use these commands:

rm -rf .crystal # to make it easier to find the .ll file
crystal build foo.cr --single-module --ll

That will generate a main.ll inside the .crystal directory (inside some other directories) which you can copy and compile on ARM.

Of course, we should be able to generate correct .o files without these extra steps, but I don't have much time to investigate right now (I'm going on vacations for a week and a half, so things will definitely slow down on my side! :-P)

[xiy]

I've been mega busy with work so haven't had a good chance to work more on this - until now!

I recently got the new Raspberry Pi 2 board, which is much easier to work with and powerful enough to handle the compilation without a sweat. I'm going to investigate it tonight and attempt a build using your comments above.

[asterite]

@xiy I hope you progress on this. Please use version 0.6.0. @jhass detected a couple of problems and we fixed them together, so it should now be easier to cross compile for ARM :-)

[ssvb]

A preliminary patch for partially working ARM support is available here. It is at least good enough to successfully compile a mandelbrot example and run the resulting binary on my ARM devboard without problems :-)

The ARM data layout configuration string is borrowed from Rust. Most likely the ARM ABI bits are also implemented correctly (at least the C structures handling passes a simple standalone test). But exceptions handling does not work right (it just segfaults), so the libunwind stuff needs some debugging. However before engaging in this activity, I'm trying to upgrade LLVM from 3.5.0 to 3.5.2 first to check if this maybe helps (the compilation is in progress and expected to take many hours on a slow dual-core ARM Cortex-A7 processor). After the exceptions handling is fixed, it should be possible to build a native ARM compiler and run the spec test suite.

[asterite]

@ssbv You are HERO!! You went ahead and implemented the ABI and modified bits of the compiler. Amazing!!

I hope the libunwind stuff works well. You can be sure we'll merge your changes once they work :-)

[xiy]

@ssvb this is awesome! I started looking at this again the other night but work has been non-stop this year and I haven't had the time I thought I would. So glad you've got this further!

I too looked at Rust and Julia when I was researching the ABI stuff so I'm glad it works. I have a week off finally so I'll see if I can give you a hand at some point!

[ssvb]

@asterite Yes, I'll try to make sure that libunwind works. First by trying simple tests with the libunwind API. And having a closer look at the exceptions handling in Rust may provide some ideas if everything else fails.

One more thing to fix is the #603 issue, because it is likely also relevant to ARM. And the Crystal code is currently full of

  ifdef x86_64
    x86_64_code              # OK, makes sense
  else
    x86_code_is_assumed_here # Not exactly great if we consider ARM and other targets
  end

@xiy Yes, the existing Crystal x86 and x86_64 ABI code has references to Rust, so it was natural to have a look there :-)

As for the new ARM ABI code, it still needs more extensive tests. I have only briefly verified that all the code branches are really taken (converting something unnecessary from Rust and having it as a dead weight does not make much sense). I initially had some doubts about the structs handling code, but then just commented these parts out in the x86_64 ABI and run it through the test suite. The failed tests provided some ideas about what kind of code can be used to test this functionality :-) Moreover, the spec tests are checking the x86_64 ABI compliance only using circumstantial evidences (by looking at substrings). This is better than nothing, but is not perfectly reliable. So I made a simple standalone custom test for structs passing/returning (based on compiling the C code by Clang and then calling it from Crystal). Now I think that it makes sense to extend this test to automatically generate a huge set of possible function argument and return type permutations in order to get complete coverage and better confidence about the ABI compatibility.

[ssvb]

I did not have much time today, but there are two news:

• The exceptions support can be temporarily stripped out from the compiler and it is still able to at least compile itself (tested just on the x86_64 hardware so far).
• There is some bug in the ARM code, which manifests itself as the frame pointer register corruption (and also crashes with blown up stack when it gets copied to the SP register). This is reproducible with non-optimized builds on the code, which uses range based string slices. The bug disappears when compiling sources as build --release. I'm trying to get a reduced testcase (usable with --prelude=empty) and check what's wrong there. Unless this is an LLVM bug, it might be some improperly handled ABI peculiarity after all.

The latter seems to be more important at the moment.

[keplersj]

As well, I've been working on and off with my own ARM stuff (ironically our code is incredibly similar.) I've been attempting to handle iOS ABI, but before I can really do anything I've started messing around with Crocoa and the X86-64 Simulator to at least build an iOS executable. I'm close but the Simulator is having the usual Apple issues, I'll see if I'm missing a step in the creation of iOS binaries that isn't needed for OS X binaries.

[keplersj]

Also, will binaries produced by the compiler with this ARM ABI run on a Generation 1 Raspberry Pi?

[ssvb]

@k2b6s9j

As well, I've been working on and off with my own ARM stuff

It's great to have more people here, who care about getting full ARM support. And iOS is one of the important targets too.

(ironically our code is incredibly similar.)

Well, as it is based on @asterite's x86 Crystal code and the ARM ABI code from Rust, some major similarities are simply unavoidable :-)

Also, will binaries produced by the compiler with this ARM ABI run on a Generation 1 Raspberry Pi?

As far as I know, the Raspbian distribution is also using the same hardfloat (gnueabihf) ABI, so it should be perfectly compatible. Just the processor is older and does not support some newer instructions, so it may be necessary to replace armv7l with armv6l (kind of like i386 vs. i686 thing on the x86 platform).

On the other hand, Android is using a bit different softfp ABI. And it's one more potential target to support.

[keplersj]

The problem with targeting Android is that you end up having to deal with the JNI. I experimented with using the JNI in Crystal not too long ago and the results were just short of disappointing. It may be possible to create a NDK library written in Crystal that Android apps could use a backend, but that still presents a few binding issues. I might see if I can use Rust's method but, I can't promise I can get it to work.

[ssvb]

Got the Crystal compiler successfully recompiling itself on ARM Linux system. But this only works for the --release configuration. The generation of non-optimized binaries is still broken. And the exceptions handling is still not supported properly.

The broken generation of non-optimized code feels a bit backwards, because one usually runs into troubles when enabling optimizations in compilers and not the other way around. I will try to use http://delta.stage.tigris.org to get a reduced testcase, because doing this in a manual way does not seem to be an easy and quick job. Maybe one day somebody implements something like C-Reduce for Crystal :-)

If anybody is interested in trying the current ARM binary of the Crystal compiler on the Raspberry Pi or similar ARM boards, I can provide it for download somewhere (together with the sources used to build it). But I hope to do something about the exceptions handling really soon.

[asterite]

@ssvb Awesome!!

About the ABI specs, right now they are hardcoded for x86_64. I think those specs should be inside an ifdef, and we should have specs for x86 and (like you did) ARM. The current specs are dummy. It's nice that you used real C code to test that they work well. I didn't want to add C files because that would maybe make things more complex, but we can have the specs write these files or just link against them without changing the Makefile or having separate files to run. I'll try to do it soon.

About the compiler crashing in non-release mode, do you have the backtrace? LLVM had some bugs related to stack unwinding, I wouldn't be surprised if there are bugs for ARM too.

[asterite]

@ssvb I just pushed a commit with specs for the x86_64 abi. For example this one.

You can do similar specs for ARM if you want. In this way we have these specs integrated in the regular spec cycle. I also like it that the C code is right next to the Crystal code, in the spec, so it's easy to see what's going on (and it's easy to add more tests).

I tried that code with 0.6.1 and it fails, so it definitely does something :-)

This will also come in handy for issue 605.

[Arcnor]

May I ask what's the status of this? Has anything changed on the latest master that might affect this? I'd really like to try running crystal on iOS / Android 😄

[zeiv]

Any chance you could post that binary, @ssvb? I haven't had much luck with with this, at least working with version 0.7.3 of the compiler. LLVM is giving me an error along the lines of "target triple not recognized" when trying to cross-compile, even though I'm using the one llc -version on my ARM machine gave me: arm-unknown-linux-gnueabihf

[ssvb]

@Arcnor iOS and Android requires a little bit more work after the regular GNU/Linux ARM is supported. And we still need to get exceptions handling working properly and find the root cause of the non-optimized build code generation issue. The current status is described in the comments above. I did not have time to work on fixing these issues yet.

@zeiv About this "target triple not recognized" problem, is LLVM & Clang configured with multitarget support in your system? For example, can you successfully crosscompile C programs?

Anyway, I'll try to upgrade both LLVM and Crystal to the current versions and provide something for download later today.

[zeiv]

Thanks for the update, ssvb. You were right, I was missing the ARM libraries and tools. I installed gcc-arm-linux-gnueabihf binutils-arm-linux-gnueabihf libgcc1-armhf-cross libsfgcc1-armhf-cross libstdc++6-armhf-cross libstdc++6-4.7-dev-armhf-cross linux-libc-dev-armhf-cross libc6-armhf-cross libc6-dev-armhf-cross and was able to "succesfully" compile a hello world program in C for ARM (it gave me a segmentation fault on my raspberry pi - but at least clang recognized the target).

file ./hello_world returns:

./hello_world: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0xfe3c789ee7adbf1539554d850c8f34f112a44c2a, not stripped

UPDATE: adding the -c flag to generate an object file and then running cc hello_world.o produces a working executable on the RPi.

Anyway, I'm still getting this error when running crystal build --cross-compile "Linux armv6l" --target "arm-unknown-linux-gnueabihf" --prelude=empty src/compiler/crystal.cr:

No available targets are compatible with this triple, see -version for the available targets.
*raise<String>:NoReturn +70 [0]
*Crystal::TargetMachine::create<String, String, Bool>:LLVM::TargetMachine +220 [0]
*Crystal::Compiler#compile<Crystal::Compiler, Array(Crystal::Compiler::Source), String>:Crystal::Compiler::Result +221 [0]
*Crystal::Command::run<Array(String)>:(Nil | File | Bool | AutoflushBufferedIO(FileDescriptorIO+) | Crystal::Compiler::Result | Array(Crystal::Init::View+:Class)) +2090 [0]
main +19431 [0]
__libc_start_main +245 [0]
_start +41 [0]

Error: you've found a bug in the Crystal compiler. Please open an issue: https://github.com/manastech/crystal/issues

[ssvb]

@zeiv I tried to build the ARMv6 Crystal compiler binary after setting a new ARMv6 rootfs, rebuilding LLVM and everything else. But appears that this build is more fragile than ARMv7 and the resulting compiler can't recompile itself (it segfaults). Moreover, the ARMv7 build of Crystal 0.7.4 can't recompile itself either. So far it looks like only Crystal 0.7.3 can be semi-usable on ARMv7 (good enough to be able to recompile itself) and I have uploaded this binary with some instructions here: https://github.com/ssvb/crystal/releases/tag/20150627-crystal-0.7.3-for-armv7

I understand that there is a lot of interest in ARMv6 because of the original Raspberry Pi. But we are not quite there yet. The Raspberry Pi 2 is an ARMv7 hardware and should be supported better.

The code generation bugs seem to be exhibiting themselves semi-randomly. It might be probably possible to try luck with other combinations of LLVM version and different recent releases of Crystal to get a self hosting capable ARMv6 build. But just one Crystal recompilation takes 20 minutes on ARM and it may take a while to do these experiments.

However the point is that we eventually want to track the root cause of this issue (in the Crystal code, or in LLVM or whatever). This means finding small reproducible testcases and debugging them.

[radarek]

What is the status of this issue? I have bought raspberry pi recently and it would be really cool to play crystal on this platform :D.

[ssvb]

@radarek Just as my previous message states, you can try an old Crystal 0.7.3 experimental build on ARM. Upgrading to the most recent Crystal version should be also possible, though the new context switching code (used instead of libpcl) may make it a little bit more difficult. The exceptions handling still needs to be implemented too.

After the initial excitement, I got a little bit disappointed because of the Crystal's gradual departure from Ruby compatibility. And haven't touched it since then. But if enough people are interested specifically in the ARM port, then I may find some motivation to finish it. And now I also got a 64-bit ARM board, which might be an interesting target too :-)

[ysbaddaden]

@ssvb strong support and motivation from me ❤️ ❤️ ❤️ ! I'd love to see Crystal running on Raspberry 3 or https://www.runabove.com/armcloud.xml for example.

Maybe the support can be done gradually. Like first have the ABI, then most of the stdlib, then have exceptions, etc. Maybe you can reintroduce libpcl for ARM only, until the required ASM is implemented.

[waj]

@ssvb If you can rebase to master I could try to get context switching and exception handling running.
I don't have much experience with ARM though. Would you recommend using a simulated environment like qemu? I could use a Raspberry otherwise.

[Sija]

👍 Running Crystal on Raspberry Pi would be nothing short of awesome!

[HCLarsen]

I can add to the interest in this idea. As a Rubyist who's a former C programmer, I love the ideas behind the Crystal language, and it would be great to be able to use it in embedded applications.

[ozra]

I'd love to use it for RPi too :-)

[faustinoaq]

@asterite Would be awesome if we could build a crystal binary for ARM 👍
Imagine: RPi, Omega2, embed systems, PLC in general.

For now I just use tiny.cr with asm, just for fun without ARM support.

But parallelism is priority, let's continue making a great language 👍
Crystal 1.0 on production for 2017-2018 maybe?

[keplersj]

I just found this great article on Docker and QEMU: https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/

It looks like Docker can be used to test ARM builds. So someone could build up the ARM ABI and test it on one machine, makes continuous integration easier too I imagine.

[ssvb]

@keplersj you don't even need Docker, just a plain QEMU works fine. My old compilation/usage instructions are still here: https://github.com/ssvb/crystal/releases/tag/20150627-crystal-0.7.3-for-armv7

[keplersj]

😄 Time to close this?

[ysbaddaden]

Done 😊

[phil294]

For anyone looking for pre-built ARM crystal packages: As per this comment, you'll find something to download over at 84codes. I just tried the one for Debian Bullseye and it works great.