C++ modules vs. GDExtension - who wins?

[Xrayez]

Context

As you may know, Godot core developers have come up with a new system called GDExtension in Godot 4.x (still in development). It's a replacement for deprecated GDNative technology for loading dynamic libraries. But could this approach replace static C++ modules in Godot? If yes, to what extent?

For reference, I've personally added a buildsystem feature in the past called "custom modules": godotengine/godot#36922. It's a really convenient feature which allows you to compile C++ modules from anywhere by recompiling the engine with them. This is something that Goost currently uses, in fact.

According to not yet fully clear design decisions, GDExtension allegedly aims to replicate what C++ modules can do. Godot even keeps track of discrepancies when comparing to C++ modules: godotengine/godot-cpp#633.

I've opened this discussion in order to see to what extent it may be reasonable to port Goost to GDExtension at some point, especially when Goost is currently advertised as a "general-purpose Godot extension", which may lead to confusion once Godot 4.0-stable is released if we decide to keep using C++ modules for whatever reasons.

Discussion

Are C++ modules discouraged to use?

According to official responses we've got from the lead developer (see for instance godotengine/godot-proposals#565 (comment)), C++ modules are discouraged to use in the future with a generic reply of "use at your own risk". Yet according to my experience, I've rarely had issues with C++ modules in Godot.

Pros of using C++ modules

Here are some reasons why I think C++ modules are superior to use at the moment:

1. No need to maintain your own buildsystem. Just use whatever Godot provides to compile modules, with ability to override any compilation flags. For now, using GDNative/GDExtension requires you to write your own scripts for building libraries, for all platforms (some of them are do not support dynamic libraries and/or GDNative/GDExtension technology). This is a lot of work. Official build scripts make it possible to compile Godot for all supported platforms as well https://github.com/godotengine/build-containers and https://github.com/godotengine/godot-build-scripts, so most of the time you don't really have to worry about maintaining your own buildsystem.

2. No need symlink plugins to be used in several Godot projects. You can just compile the engine with modules and have all classes available built-in. Really, this is what "batteries included" means. This is not possible to achieve with GDExtension at the moment. At most, only global editor-only plugins are going to be allowed (like accessing from editor://, see Universal plugins [wip] godotengine/godot#35432 (comment)), but not much more than that.

3. GDExtension will be limited to what ClassDB provides. If something is not exposed, you have to make changes upstream (Godot) to resolve such limitation. But then your use case may be considered as a corner case which is not worth to be exposed/maintained in Godot. Under those circumstances, it's much easier to add a feature via C++ modules because you literally have access to mostly everything in the engine. You can also patch the engine's own code slightly to resolve limitations, and features like Add callbacks to Main::setup() / Main::start() / Main::cleanup(), so to allow much freedom to modules. godotengine/godot-proposals#1593 become possible to implement. This gives back control over code.

4. Never-ending discrepancies between GDExtension and the engine. For Godot Engine contributors (like me), this will result in a delay of development. If some feature becomes available in the engine, it will always take some delay before repositories that provide GDExtension implementations like https://github.com/godotengine/godot-cpp catch up.

5. Godot's own source code is like a database of existing solutions. Data structures can also be reused extensively without having to use C++ standard library (which may bloat binary size and make compiler errors harder to read) If you develop a C++ module, in a lot of cases you can just copy-paste the code and it will work with little to no modifications. This really speeds up the development.

Pros of using GDExtension

1. Faster to recompile. Building the engine is a slow process, but frankly not that slow on my machine (10 min - full rebuild, 15-30 seconds - single file change).

2. Easier adoption for users. It might be easier to download and install a library rather than downloading Godot engine with Goost module precompiled.

3. Somewhat independent development. Those people that use Goost rely on releases of Godot + Goost. If Godot makes a new release but Goost remains at the same state, one has to make a new Godot + Goost release anyway.

I think some or most of the above limitations can be resolved, but Godot core developers must show seriousness in maintaining such a system on par with the engine itself.

Issues

There are already limitations that I see could be problematic if we port Goost to use GDExtension, so below is the list of concrete examples of issues:

• AbstractPolygon2DEditor is not exposed in ClassDB: https://github.com/goostengine/goost/blob/173af73e45395fea2c6a9ada6c78f71941f71448/scene/2d/editor/poly_node_2d_editor_plugin.h. Goost extensively reuses this class for PolyNode2D used for boolean operations in the scene (and much more). If this class is not exposed, it would mean maintaining our own implementation, not good - duplication, too easy to go out of sync implementation-wise.
• No way to extend Texture2D according to [GDExtension] Missing methods, classes and API discrepancies. godotengine/godot-cpp#633 (comment), which is needed for classes like GradientTexture2D, LightTexture etc. Image API is vital in general because Goost has a dedicated image component.
• Some virtual methods could be considered for "internal use", even though they are public. For instance, see https://github.com/goostengine/goost/blob/eb27535042f8c89f6f81acc192909baabc0e939d/scene/2d/poly_shape_2d.h, namely add_child_notify() needed to manage children for classes like PolyCollisionShape2D to rebuild shapes without polling or using get_parent(), which is in most cases discouraged.
• Working with Variant internals might be problematic. Goost has VariantResource, for instance. See Godot proposals like Expose Variant's internal generic pointer for GDNative godotengine/godot-proposals#3037.
• No access to some of the engine's callbacks, see for instance

  goost/core/register_core_types.cpp

  Line 32 in 1e2074e

  SceneTree::add_idle_callback(&GoostEngine::flush_calls);

  needed to make unique deferred calls in the engine using GoostEngine.defer_call_unique(), which is useful for optimization that typically use "dirty" flag.

Issues in other projects

• Porting to GDExtension Zylann/godot_voxel#333

Conclusion

It might be a bit early to make final conclusions, but I'd like to track issues and limitations with GDExtension here, and see if it's worth porting Goost to use GDExtension as a whole, or perhaps still provide Godot + Goost builds, or a combination of both. But according to issues outlined above, it might not be reasonable to port Goost to use GDExtension mainly due to the nature of Goost, which actually aims to reuse and extend Godot's core. This potentially signifies that more limitations like this may popup in the future.

You also have to be aware of Goost development philosophy.

This post will be updated as development goes on, with more concrete examples. Feel free to comment in the meantime!

[nonunknown]

I took a read and here's my thoughts:

• First, I think GDNative and Modules where tools for archiving different things, like Modules you can bring other libs, extends the engine in a depeer way, while GDNative you can do more basics things, like heavy calcs and implement game mechanics, which you know, makes sense! Now GDExtension seems to be different from GDNative, because its like a portable version of C++ Modules!

• As you stated in Pros of GDExtension, topics 1 & 2 already "crushes" Modules, because the end-user just want to get things done in a simpler way as possible, also I think its the best way for people who want to get into C++ (or whatever language) he wants to learn and using his favorite game engine.

• The only Con I've found by messing with GDExtension is that you cant recompile your code with the engine open, like it was in GDNative see [GDExtension] updating library during editor opened, crashes the engine godotengine/godot-cpp#638

• The biggest hassle that I can think of is the compilation for all platforms, thats bother me a lot!

[Xrayez]

Yes, I realize that the two could be used to achieve different things, therefore perhaps using both C++ modules and GDExtension may be worth it. But as you say, GDExtension/GDNative would make more sense for implementing specific game mechanics, which is something that might be slightly outside of scope for Goost. Due to this, perhaps this is where game-specific community extensions can be developed that build upon both Godot and Goost core functionality... And in order to achieve the "batteries included" experience, all those extensions will need to be bundled by default.

At the end of the day, it's also about personal motivation... For the projects I work on, I have to recompile the engine anyway, as certain limitations cannot be worked around via GDScript nor GDNative, so it just doesn't make sense to use GDExtension, and those use cases cannot be contributed to Godot due to those use cases being treated as too specific. To put shortly: GDExtension is easier for users, but more difficult to manage for developers!

• The biggest hassle that I can think of is the compilation for all platforms, thats bother me a lot!

This is the biggest pain out of them all in my opinion. For those build scripts to be maintained reliably by Godot developers, some of the built-in modules have to be ported to officially maintained GDExtensions (IIRC, that would be video decoder extensions). Otherwise I'm afraid that this is the part that might be neglected.

Godot's own buildsystem is constantly maintained because it has to be maintained for the engine to work. The same kind of nurturing should be applied to extension development.

[nonunknown]

well I cant see where GDExtension is difficult to maintain (at least on the end-user side) because you can separate every class by a compiled dll/so file! in the end its easier to mantain, the problem is that you cant debug while you can in modules!

[Xrayez]

Also, I've been thinking of introducing the notion of experimental features in Goost. The idea of experimental features is that they:

• might not be actively maintained
• can be still in development, therefore may contain bugs
• can be safely removed in the future if a feature is not that useful (or too specific) 😛

This removes the obligation of maintenance, while also allowing for even faster development cycles. I've never actually seen that sort of experimental development in Godot.

And this is where GDExtension can be incorporated for those kind of experimental features safely.

[nonunknown]

this is cool!

[ghost]

I am going for a whole different approach.

Instead of beating the same old dead horse I have been looking for a new language to implement that will be faster and more multi-purpose than the currently available languages.

First I looked into Wren and that had some issues. Is very limited in scope and the truthy/falsy is backwards and other issues. Then I was gonna go with a few other ones and spent a bit of time until I found what I think is the best one for Godot daScript https://dascript.org/. Compatible licensing model, is strongly typed, and looks similar to GDScript. According to their benchmarks is faster than LuaJIT, faster than C# and Java, and it reaches equal performance to unoptimized C++. Even more interesting it has pointer types like C++ and can even natively interface with external code at binary level using unsafe keyword I believe kinda how C# does it so built in abi.

This language can potentially replace GDScript, Modules, GDNative and even C#. The beautiful thing is that you will be able to write all the code right in Godot Editor so you will no longer need compilers since is an interpreted language. You can build libraries of addons using .da scripts that can do anything a module, GDExtension/GDNative can.

[ghost]

Nim is compiled though. DaScript is interpreted plus it can already do C/C++ interop since it can do pointers. Is a scripting language with C++ pointers that doesn't need to be compiled and runs as fast as C++. Also I don't expect anyone to implement it. I will implement it myself if I get that far. @insomniacUNDERSCORElemon

[insomniacUNDERSCORElemon]

Nim is compiled though. DaScript is interpreted plus...

I get it. Just adding that as I couldn't find 3rd-party info (reviews etc) and am not a programmer anyways so much is lost on me.

That, and Nim-lang offers a lot of power. For instance you can run it as a script even in compiled projects (see Enu by dsrw that has live-coding and a custom DSL). I don't know if it's as fast as C++ when ran as a script, but it's an option.

[ghost]

That, and Nim-lang offers a lot of power. For instance you can run it as a script even in compiled projects (see Enu by dsrw that has live-coding and a custom DSL). I don't know if it's as fast as C++ when ran as a script, but it's an option.

That's not Nim language . That is a simplified rule based language for kids called Enu that's inspired by and built on top of Nim. Still requires Nim version 1.6.0. Nim is a systems language not a scripting language. It requires compiling and produces machine code.

Nim is 0-50% slower than C++ compiled
GDScript is about 200-500% slower than C++ parsed to opcode then opcode interpreted
DaScript is 0-70% slower than C++ AoT

Now you not gonna find too much about DaScript but I can tell you what I know. It was designed to be used for video games and to be very small, ultra fast and easy to embed. Scripts get parsed straight to machine code kinda like C++ except unoptimized naive machine code and this is what actually runs. The language can do anything C++ can but doesn't need to be compiled because it's intended use is for scripting not system software.

[insomniacUNDERSCORElemon]

You can run nimscript just by giving your nim file a .nims extension. In a compiled project it seems you need to add code to configure it, but you can just copy Enu's code that does this/do something similar. Nim's DSL capability would be useful for writing stuff for an engine (could probably get close to GdScript). Nim also has REPL implementations.

Although there's also no strict need for non-compiling in most cases anyways, given you could easily compile when running the project... especially if using TCC or Clang (--opt:size will give a decent performance boost for not too much compilation cost (though I haven't tried a large project), especially comparing to no-opt GCC). Even Godot's remote scene tree might not be an issue for non-code stuff (for code, using a watcher and re-compiling mid-scene might even be possible too).

For DaScript I'm specifically wondering if it could do things like AI/simulation/networking etc... not just with speed (processing and latency) but also language capability.

[azur-wolve]

@insomniacUNDERSCORElemon
there are plans to, in some future, add to GDScript the possibility of pre-compile it through LLVM to C-generated machine code.
It meab that the workflow would be iterate fast (prototyping, testing, etc) in the interpreted way, and eventually pre-compile it for release with C-like performance.
They will not drop GDScript.