Modularize the crystal-compiler

[mdedetrich]

Would it be possible to modularize the crystal compiler (i.e. make shards out of the compiler internals which the crystal compiler itself would require as a shard)

The reason behind this is I have a couple of projects in mind, one is a different compiler backend for crystal (which means I want to reuse the crystal parser-compiler) and the other is the ability to include the crystal-compiler itself in crystal applications (i.e. so you can use crystal as a configuration language)

[RX14]

You can simply require the compiler, or parts of the compiler, from any crystal install: the compiler sourcecode is part of the stdlib. Simply require "compiler/crystal/<whatever>" and code away.

[mdedetrich]

Understood, however you I would need to add entire crystal repo as a git module and manually require it, as well as managing the dependency of this git module seperately

In other words, its not ideal (but its doable)

[RX14]

Why would you need to add the crystal repo as a git module? I just said it was part of the stdlib and compiler sourcecode can be required anywhere with even less work than using a shard.

[faustinoaq]

@RX14 Would be posible to use a different backend for Crystal?

I mean, an alternative to LLVM, maybe a JS backend like Nim or Kotlin.

Currently, JavaScript world is very important.

Related: #3634 (Web Assembly would be useful but still will use LLVM)

[refi64]

JS backend would require more work than a WebAssembly backend...you still have to implement another stdlib, only now you're playing by the JS rules and have to write another code generator. Things like the struct vs class distinction would be irrelevant, and I'm not even sure how the Int32 vs Int64 deal would work.

[faustinoaq]

So, WebAssembly is the way 😅

[straight-shoota]

Currently, JavaScript world is very important.

One day Crystal will obviously become more important than JavaScript. 😃

[mdedetrich]

Actually the idea is to make a JS backend, and surprisingly a JS backend is easier than a WASM backend (this is the experience that the Scala guys had with Scala.js, im actually coming from Scala and exploring other languages)

WASM has a very unique way of handling memory management, furthermore it doesn't have any interaction with the DOM so its incredibly limited for now.

Why would you need to add the crystal repo as a git module? I just said it was part of the stdlib and compiler sourcecode can be required anywhere with even less work than using a shard.

The compiler itself is part of the stdlib?

[faustinoaq]

The compiler itself is part of the stdlib?

@mdedetrich Yes, It's

[refi64]

Actually the idea is to make a JS backend, and surprisingly a JS backend is easier than a WASM backend

Crystal uses LLVM, which would mostly take care of the actual codegen.

[RX14]

surprisingly a JS backend is easier than a WASM backend (this is the experience that the Scala guys had with Scala.js, im actually coming from Scala and exploring other languages)

Is that coming from something that emits LLVM IR already or not? Crystal's entire codegen phase is already emitting LLVM IR, surely using LLVM's WASM output will be easier than outputting JS. If scala had to port to LLVM and make it work with WASM that sounds like an entirely different proposition to me. I don't know anything about scala's position though so that's all a guess.

[mdedetrich]

Is that coming from something that emits LLVM IR already or not?

Scala already has scala-native, which has an LLVM backend

Crystal's entire codegen phase is already emitting LLVM IR, surely using LLVM's WASM output will be easier than outputting JS. If scala had to port to LLVM and make it work with WASM that sounds like an entirely different proposition to me. I don't know anything about scala's position though so that's all a guess.

Afaik, although Scala.js requires a different backend, its not that complicated because

1. There is only single threading (no multithreading involved)
2. Don't need to worry about GC at all
3. Scala's semantics (and I assume Crystal's as well) maps fairly well to Javascripts semantics
4. Google Closure Compiler helps a lot when it comes to optimization. The scala-js backend generates JS that is very easily optimized by GCC, which means that Scala-js doesn't need to do as much optimization by itself

Of course, the main point is that the future of WASM (or when it will get there) is unclear at this point. In any case this is just something that I personally want to look into ;)

[ysbaddaden]

The idea, if I understand correctly, is to create a JavaScript transpiler, bypassing LLVM completely. That is take the semantically analyzed crystal AST, and generate JavaScript instead of generating LLVM IR to eventually generate WASM (or anything else).

I wouldn't be surprised for such a solution to be simpler to handle than WASM; granted that you'll may not have access to low level features (pointers) or the crystal stdlib, but you gain direct access to the whole JavaScript land —which seems impossible currently in WASM.

With a crystal compatible runtime, and a previously agreed subset of (in)compatibility (some things possible in crystal aren't possible in the transpiler) you could end up with something similar to Opal (a Ruby engine that transpiles to JavaScript).

That's still hard work, though. But the most interesting part of the crystal compiler (lexer, parser, semantic analysis) can be reused, yes.

[mdedetrich]

The idea, if I understand correctly, is to create a JavaScript transpiler, bypassing LLVM completely. That is take the semantically analyzed crystal AST, and generate JavaScript instead of generating LLVM IR to eventually generate WASM (or anything else).

I wouldn't be surprised for such a solution to be simpler to handle than WASM; granted that you'll may not have access to low level features (pointers) or the crystal stdlib, but you gain direct access to the whole JavaScript land —which seems impossible currently in WASM.

Yup exactly

With a crystal compatible runtime, and a previously agreed subset of (in)compatibility (some things possible in crystal aren't possible in the transpiler) you could end up with something similar to Opal (a Ruby engine that transpiles to JavaScript).

Scala.js didn't have this problem, they pretty much support everything apart from Char and certain exceptions (for interest you can see them here https://www.scala-js.org/doc/semantics.html)

[RX14]

Surely the better idea would to use the already-mature emscripten to port the stdlib. Then you wouldn't have to fight both at the same time. The counter-argument is that translating crystal to more semantically correct JS will be easier to debug and easier to minify. Also having 2 possible routes to generating JS will make the stdlib port more general.

Also can we not use gcc to refer to the google closure compiler, overloading acronyms is terrible.

[mdedetrich]

Surely the better idea would to use the already-mature emscripten to port the stdlib. Then you wouldn't have to fight both at the same time. The counter-argument is that translating crystal to more semantically correct JS will be easier to debug and easier to minify. Also having 2 possible routes to generating JS will make the stdlib port more general.

Actually there are more compelling arguments to go for a JS backend

• To be considered at least somewhat "successful", you need to interopt with the Javascript ecosystem. This is impossible with WASM (furthermore this is evidence by the success of projects like Elm, Typescript, Clojurescript and Scala.js)
• Porting the current Crystal stdlib is going to create discrepancies for the platform. Its actually better off to create a stdlib specifically for the platform, and then make a compatibility layer ontop (if needed)
• Can't interact with the DOM at all, half the reason why things like Scala.js is popular is because there are wrappers for things like React (i.e. see https://github.com/japgolly/scalajs-react or https://github.com/omcljs/om). Would love to see a crystal DSL for interacting with things like React
• Can interface with things like Electron to provide UI's (good short term solution to GUI's in crystal)

[faustinoaq]

Since #3634 the opinions in this issue are nice to read.

@mdedetrich A React wrapper is awesome, Nim also has it https://github.com/andreaferretti/react.nim

Would be awesome to see Crystal.js compiling a React project written in Crystal 😄

JavaScript is like assembler for browsers right now 😅

[RX14]

@mdedetrich emscripten != WASM. They both compile to JS and as such can make arbitrary JS calls. GC is an issue though, and it's probably a big enough issue to create a custom codegen phase.

Creating an entirely seperate stdlib may be an easy way to start a compiler port but there's no point in merging support unless it supports at least a core set of the stdlib.

Regardless, this issue has become off-topic, would it be better to discuss a JS backend in a dedicated issue?

[faustinoaq]

@RX14 Maybe, JS backend issue already exist, see: #829 (Can be reopened?) and #535.

[RX14]

@faustinoaq #829 is about wasm, #535 is about asm.js. Neither of those are applicable to writing a backend without LLVM involved.

[mdedetrich]

Creating an entirely seperate stdlib may be an easy way to start a compiler port but there's no point in merging support unless it supports at least a core set of the stdlib.

Actually the idea is to make a seperate compiler for the .js backend rather than to merge it into the main crystal-lang compiler as there are too many differences to consider

Regardless, this issue has become off-topic, would it be better to discuss a JS backend in a dedicated issue?

Agreed, should I close the issue?

[RX14]

@mdedetrich if you make a new issue, sure close this one. The compiler is already reasonably modular and usable from other code.

I don't think creating a seperate project is a good idea. We should aim to use the same compiler and same stdlib for every project. Creating a seperate project for testing is fine but you should aim for it to be mergeable. That means upstreaming some patches in the parser and semantic passes if it helps you.

[faustinoaq]

I don't think creating a seperate project is a good idea

Ahhh, I had a logo 😅 (just kidding)

@RX14 You are right, duplicate eforces are bad, maybe we can compile to javascript using a flag --js like Nim does. However, languages like Kotlin and Scala have a separate project to achieve a JS backed.

Some related projects I found on http://crystalshards.xyz/:

• https://github.com/jessedoyle/duktape.cr - Evaluate Javascript from Crystal
• https://github.com/geppetto-apps/crow - Transpile/compile Crystal to Flow/JavaScript

@mdedetrich are you going to open a issue related to JS backend? 😅

[RX14]

Crow looks like a good start. If it could be brought up to latest crystal and working on nearly all crystal syntax, it would be fantastic. Working out how to take advantage of the huge repository of existing compiler specs to test the new codegen is a must. After that, a stdlib port could be attempted.

After it works, we could start thinking about whether we want to merge this and if so how. Having 2 entirely separate backbends to the crystal compiler would be great for keeping clean code, as it helps ensure separation of concerns between parts of the compiler.

[asterite]

I think this issue went a bit off-topic :-)

Closing.