Support for Other Languages?

[stellarpower]

Nice architecture! Shows what can be done.

Given everything is modular, is there much chance of supporting other languages? Don't know if it would be a lot of work, but an (almost) language-agnostic framework would be really nice. I am thinking Rust would be an obvious one, perhaps to wrap a whole crate.

[Papierkorb]

Hello @stellarpower !

I've been long out of the Crystal game.. Anyway, that's what SWIG is basically capable of (In theory, not that it can do it today).

I guess that you could recycle the architecture of bindgen, and parts of some modules. Parsing C++ was really finicky, I guess that Rusts tooling is much better in that regard. Still, that's barely half the battle, the (semi-)automatic mapping is much harder. On the upside, Rust is much simpler than C++, if only by not having accumulated as much cruft.

Nowadays I'm still doing C++ for the most part .. and the other part Rust, actually. And I actually had ample projects where I could actually apply the knowledge gained through bindgen in the industry, so it's surely not all in vain ;)

If you were to take such an undertaking (Even if just for educational purposes), here's what I'd do:

1. Fork bindgen and throw out like half of it (The really C++ specific parts)
2. Get a rust parser to work which outputs a format similar as the current Clang-one does. That is, output as much information as useful (not as possible) in JSON. Every bit complexity you introduce here will have ripple effects to every line of code you write later on.
3. Start small, write the code to generate calling a simple function like add(x: i32, y: i32) -> i32. This is also where you will meet the type system of bindgen. Ensure to understand it thoroughly - You should be able to greatly simplify it, as C++ does crazy things Rust doesn't. Again: The type stuff has ripple effects. Add as many convenience functions here as possible, a few highly complex pieces is much easier to maintain than scattering those complexities across the code base.
4. Map structures next, maybe simple C-style enums too. This gets finicky fast (You're seeing a theme yet?), so start with #[repr(C, packed)] structures.
5. You'll at latest now want to think how you map Rust-style enums over. And especially: Make them fun to use in Crystal!
6. Extend the static run-time code to map Option, Result, et al over in a Crystal-idiomatic fashion. Same applies here: If the generated code sucks to use, then no one will use it.
7. Still not tired? You still need to adapt traits and their impls. You may have already added initial support for this earlier, now it's the time to patch things up.
8. Things will break later on. Write unit-tests. Annoying, but less so than playing what feels like Jenga all the time.

Honestly this seems daunting. I'm not sugar coating it. If you still think this sounds like a good project, please try it! I learned a lot about about code generation, and translating things that weren't meant to be translatable (Doing fun things in the process, those where everyone says "Don't try them at home"). Last but not least, I attribute my knowledge about Software Architecture to complex projects like this. It's a great learning experience, and for that alone, if you have the will and time, I say: Go for it!

[stellarpower]

Thanks for the detailed reply! Appreciate it.

I have been out of the Crystal game a bit myself, can be hard to write projects in it sometimes, without starting and then finding you need a library that isn't available.

So, all this probably more than I have time for - and crucially, I don't actually speak any Rust. I may learn some some day but I've only dabbled thus far, so main motivation for asking, beyond the hypothetical, was the point of view of benefitting from using some crates. If google carbon or something like that comes along in the next few years, I probably would spend the time learning that through and through, as I think modern C++ with all the legacy bits dropped and full backwards-compatibility would be a hell of a language.

FYI magnus may be useful for a few of those items you enumerate above. If they already have the logic of mapping Rust types to Ruby, then whilst the backend would be very different, the semantics could probably be immitated.

I attribute my knowledge about Software Architecture to complex projects like this.

I agree, I think working at levels like this gives one a fundamentally different view on programming. I think sometimes people write their projects and like keeping the code very "simple", and would always think my approach is overcomplicating things. But code you can perhaps read very quickly then ends up being lots of copy-paste, doesn't scale well and breaks easily. All the stuff you read about in books when learning about breaking it up into small steps and chaining it together really is true as soon as you are working on a large-ish codebase doing something sufficiently complicated. You have to architect it, just throwing the idea out onto the page in a terse but hardcoded form that solves today's exact problem won't cut it.

I wonder if clang has evolved at all since you first attempted this. I know the folks at ROOT merged cint into cling using clang's modular architecture, and last time I looked at that project, they were a way behind the clang/LLVM mainline, but, were gradually planning on merging more back and forth between the projects to reduce the duplication and bring clang itself closer to being able to run the interpreter. I'd guess the basic parsing classes etc. haven't changed enormously, but you'd think it'd get more language-agnostic and more self-contained as time moves on. From memory, I thought the rust compiler used LLVM, but looking at it I guess the whole parsing architecture here wouldn't be re-usable. Unless there's some way to output an AST from a crate that does just that, but still many holes to fill in.

Anyway, thanks a lot! I am unlikely to attempt this, but it is interesting to think about nonetheless.