[WIP] Add ASM.JS/Emscripten Target #3634
Conversation
Using a few tricks basic Crystal programs (like single lines containing numbers basic) are being compiled to JavaScript.
* master: (69 commits) Fixed: splat restriction and double splat restriction didn't match empty tuple/named-tuple Fixed crystal-lang#1203: allow using free var to match the size of a static array Updated Changelog Compiler: don't remove old directories when compiling macro run programs. Related to crystal-lang#2625 Fixed crystal-lang#2624: Running multiple macro runs doesn't work Docker: Preinstall git in released docker image Update and fix TypeNode macro method docs Fixed wrong datatype in 0.17 branch Docs: use `*` and `**` before type argument for Tuple and NamedTuple Docs: use `*` and `**` before type argument for Tuple and NamedTuple Compiler: allow a double splat restriction on a double splat argument Compiler: allow a splat restriction on a splat argument Compiler: allow restriction in double splat Compiler: guess type from captured block without restriction Fixed crystal-lang#731: initialize's default arguments are incorrectly evaluated at the class scope Fixed crystal-lang#2619: wrong codegen for global variable assignment in type declaration Compiler: allow a double splat restriction on a double splat argument Compiler: allow a splat restriction on a splat argument Compiler: allow restriction in double splat Compiler: guess type from captured block without restriction ...
Cuts down on a dependency, and I believe the JS engine will do GC.
This reverts commit 61b682a.
Inspiration taken heavily from Rust.
This reverts commit cc2c0f8.
This can be used with emscripten by setting LLVM_CONFIG to emscripten’s embedded llvm-config
I tried generating libc using posix but ran into issues
Needs testing
|
I'm not sure how to handle garbage collection at this point. Everything seems to compile, although it is a bit convoluted and needs some work. First you must compile the compiler with From here you can begin compiling Crystal files to JavaScript, by setting With a basic def number
1
end
numberRunning From here we have to compile the emitted LLVM IR code to This produces a 37,000+ line JavaScript which runs in node and errors out:
From here anyone should be able to help out and hopefully get this merged into master. I don't have a lot of time to dedicate to open source projects at the moment so it would be really appreciated if someone(s) interested in this picked up it from here. |
|
/cc @catmando |
|
@kripken any advice on how Crystal should integrate Emscripten? |
|
I'd love to help with advice if I can. I'm not that familiar with how Crystal is architected, though. Anything specific I can answer? In general, it looks like you're generating LLVM IR using one LLVM (Crystal's? Or plain LLVM?), and then loading the IR in text form in emscripten's LLVM? That can sometimes work. But the triple difference can cause problems. You should use the wasm triple for this, as it is the closest to the asm.js triple (and we are working to make them as close as possible), instead of your native system's triple. |
|
Why does the compiler need to be compiled by emscriptem? I'm figuring you are cross compiling using emscriptem as the back end only |
|
@catmando The compiler isn't being compiled by emscripten. It's being told the location of emscripten's |
|
@kripken Crystal uses a plain LLVM installed on the system to generate LLVM IR which it then passes to Clang to produce a binary. This PR allows Crystal to use the LLVM in Emscripten to generate LLVM IR and then pass it to Currently it appears that a Crystal program and the Crystal standard library can be compiled to JavaScript, but the programs are missing their runtime dependencies. Most Crystal programs are dependent on libgc, libpcre, libevent, and libpcl. I'm not quite sure how we should account for these libraries when using Emscripten. Usually the user installs them on their system an Crystal links against the libraries, but that doesn't seem to an option with Emscripten. |
|
@keplersj: oh ok, then if you are using the same triple everywhere that should be fine. Regarding libraries, yes, they need to be ported to emscripten to be usable - a JS program can't link with local system libraries that contain native code. Generally you would port them to emscripten, generating a bitcode file for each, then link that with the program's bitcode, then compile all of that into JS. One possible issue with libgc in that list is that it needs to scan the stack for roots. That's not possible in JS. One solution is to take advantage of the fact that JS code needs to run in short-lived events anyhow, due to how the web works, and so you unwind the stack completely from time to time, and it is safe to GC then. (The only downside to that is if in a single such frame of execution you need to GC in the middle in order to not run out of memory.) In wasm this might improve in the future. |
|
The best place to start here is using a js-specific prelude. Start with |
|
I'm a total newbie regarding emscripten and asm.js, so with this question I just want to learn/understand. What would be a use case for compiling Crystal to JS? |
|
@asterite sharing code between a crystal backend and frontend would be the obvious answer. |
|
@asterite You could run crystal code natively in the browser, (almost) without JS |
|
@asterite JavaScript runs in more places than Crystal does at the moment. Compiling to JS would allow Crystal code to be run on Windows, iOS, Android, Pebble, really any platform with a JavaScript Engine. |
|
Maybe that could lead to:
|
|
@drhuffman12 although it would be cool to write crystal on the client, the playground could never be serverless because there are some things that crystal can do that can not be replicated in a browser (e.g. spawning a process) |
|
One should distinguish between these different goals:
3 could make crystal a nice new standard language to do fast stuff in the browser that JS can't do I think there is precedence for custom script tags with embedded mruby/python. IMHO we should drop JS from the discussion, just look at the wasm DEMO and be amazed. Exciting times. |
|
Despite being fun to achieve, I'm afraid it has the same problem than trying to target AVR (Arduino) or an ARM cortex M0: it's easy to target them with an empty prelude, but if you can't have BOEHM-GC running (at least) then you can't use anything from the corelib and stdlib. I'm not even talking of POSIX requirements, but you can barely use pointers or strings. Note that compiling to JS (emscripten, wasm, whatever) may only make sense when targeting the browser. Compiling Crystal to JS to run on Node to run on Android... Nonsense. Let's target the platforms natively instead :-) |
|
I think you can actually use the Boehm GC under emscripten: https://groups.google.com/forum/m/#!topic/emscripten-discuss/SGwr56gXCZo |
|
@kirbyfan64: yes, Boehm works, the only limitation is stack scanning as mentioned in this comment. The workaround mentioned there is good enough for it to be used in production, but you do need to be aware of it. |
|
I would think a PR like this would take some time to merge. Would it make sense to pull changes that are not |
^ Interesting 😅 |
|
@keplersj Thank you for this! But since this is WIP and not finished, there really isn't much we can do here. Instead of leaving this PR open, we'd like to close it to avoid clutter. Please reopen or send another PR if it's feature-complete. Thank you! |
Both emscripten and wasi, for now. Though will likely only keep one. Based on crystal-lang#3634
Both emscripten and wasi, for now. Though will likely only keep one. Based on crystal-lang#3634
Both emscripten and wasi, for now. Though will likely only keep one. Based on crystal-lang#3634
Both emscripten and wasi, for now. Though will likely only keep one. Based on crystal-lang#3634
Much like #3582 this is very initial.
Fixes #535