Porting to GDExtension

[Zylann]

This issue tracks the state of this module in regards to GDExtension.

It has been a long term goal to port this module to a dynamic library that can be shipped as a Godot plugin, which would allow using official Godot versions instead of having to recompile the editor and export templates. This makes distributing way easier through the asset library.
However it wasn't done in Godot 3 because GDNative had annoying issues in its design, but in Godot 4 it was rewritten as GDExtensions. This gets rid of some of the design problems, and the engine devs want it to be a true alternative to actually porting modules for real.

There are, however, still issues with it at the moment. They are not necessarily blockers, but if solved, would make maintaining and using the library way easier.
The list is very long so I made a separate issue to track them

Once enough issues are solved, other changes will happen in this repo.

At the moment, the project aims to support compiling both as a module and as an extension. The main reason is the time it takes to fix issues with GDExtension, so the working module should remain unaffected.

Project-wide changes

Repository restructuration

Porting to GDExtension will make the module a plugin. That means the file structure will change, as there will be a few more things to include to it (helper scripts, shaders...), and it would be nice to not have all the folders about different things all at the same level. The C++ part will not change too much, but it will be moved to a lower-level folder. This structrure is not set in stone, it's mostly a draft.

.git/
.github/
bin/
    libvoxel-editor.dll
    libvoxel.dll
    ...
cpp/
    .gdignore
    terrain/
    meshers/
    util/
    ...
    thirdparty/
        meshoptimizer/
        godot-cpp/
        ...
    SConstruct
assets/
    icons/
    shaders/
    ...
doc/
    .gdignore
    source/
    ...
misc/
    .gdignore
    check_ci_result.py
voxel.gdextension
README.md
LICENSE.md

Note: contrary to many other addons, this structure will not start from the root of a Godot project. Everything will be the contents of what would be inside res://addons/zylann.voxel/. This makes it easier to submodule in other projects. The AssetLibrary package will still contain a full hierarchy.

Source build integration

I have a personal project which I would like to develop using GDExtension as well, which means depending on voxel as well. Usually if your GDExtension depends on another, you would have to first add the extension, recompile bindings including extra extension symbols and only then build your own GDExtension.
This is cumbersome because I might very likely keep developping this engine alongside my project (frequently modifying it), and I want to have C++ level access, instead of a limited "Godot-compilant" API access.
This is why I have the idea that this extension should be integrable as source in another GDExtension, such that the voxel engine and my project can compile as a single library. This might boil down to some features in build scripts.

Progressive transition: supporting both module and GDExtension

Because of a lot of uncertainty, porting the entire module in one go is not a good idea. Doing a bunch of porting and ending up "stranded" in a half-working state is not desirable. Instead, it would be better to start making it able to compile both as a GDExtension and as a module. Then, eventually shift the main development into GDExtension as issues get fixed.

Achieving this goal comes with a number of caveats on top of those already listed earlier:

• Header includes in GDExtension are different than Godot core. This can be worked around with proxy headers under util/godot/, which can conditionally include the right header depending on the compilation target.
• Godot core does not have the godot:: namespace. A workaround is unfortunately to either #define godot or to use namespace godot in all header files. This might cause a number of name clash issues which we'll have to figure out (unless Godot core decides later to namespace its code, of course). This module does not intend to be part of Godot's core classes and uses its own namespace, therefore it does not define its classes inside godot:: either. Another possible workaround is to re-define Godot classes with typedef or using inside a godot{} namespace, and use the aliases exclusively.
• Function signatures are different. Some classes extend Godot ones and implement virtual functions. Signatures and even method names can differ in GDExtension (starting with _, Variant-compatible parameters only, const Ref<T>& on all refcounted types...). I can't think of any other way than using #ifdef.
• _notification is different: I believe GDExtension exposes it as a virtual method, while in core this is not the case and rather gets picked up magically by the GDCLASS macro?
• SCons build scripts may work differently. An obvious one. The main difference is, building as a module makes the voxel engine a slave "puppet" of Godot, while building a GDExtension makes it a "master". Ideally they should be as similar as possible. The common part specifying how the voxel engine builds may be in its own script, and we should keep only the different parts separate, with as few repetitions as possible. The C++ bindings should ideally be seen as just a regular library dependency. It should also be noted that some dependencies like FastNoise2 will likely switch to dynamic libraries compiled with CMake in the future, because it is easier to maintain them this way (while doing this as a module was impractical due to compilation flags and means of distribution).
• memnew has limitations in GDExtension: I think I recall that while memnew was made to work similarly to Godot core with Godot-derived classes in GDExtensions, it wouldn't work with arguments or with non-Godot-derived classes. The module is using memnew on non-Godot types in various places, and also on custom UniquePtr and std::shared_ptr definitions (mostly to use the same memory allocator). This might need some wrapping up to make code work with both situations.
• More to come eventually...

It should also be noted this module already tends to avoid depending on Godot in some areas. It uses standard library containers and defines its own threading classes and logging macros for example, which can be used in place of Godot alternatives. These will work in either configuration as long as they are defined/implemented once properly.

[BastiaanOlij]

No separation of editor and release libraries: just like the editor comes in two versions (editor and export templates), libraries must be able to do this too. However I don't recall GDNative having that feature. This may need a second check in GDExtension.

This one should be resolved, debug builds are needed for the editor, release builds can be released with the end result. You do so like this:

[libraries]

linux.64.debug="res://addons/godot-openvr/bin/x11/libgodot_openvr_debug.so"
linux.64.release="res://addons/godot-openvr/bin/x11/libgodot_openvr_release.so"
windows.64.debug="res://addons/godot-openvr/bin/win64/libgodot_openvr_debug.dll"
windows.64.release="res://addons/godot-openvr/bin/win64/libgodot_openvr_release.dll"

Note though that currently exporting extensions is broken, you have to copy the library in place manually

[nonunknown]

@Zylann In case you need, I've made some scripts to accelerate the development with gdextension, in case you need something to ease it for you just tell me:

https://github.com/nonunknown/gdextension-creator

[Zylann]

After a week of work, I just finished introducing a new build target in the gdextension branch. This is only a first pass, so it should compile on Windows, but it is completely untested for now (so will likely print errors and eventually crash the engine).
If you want to have a look at how this is achieved in code: https://github.com/Zylann/godot_voxel/tree/gdextension

Some issues are tracked in the source code of this project as // TODO GDX: [...] comments.

The list became very long so I created a separate issue: #442

[IPlayKindred]

any updates on this? how far is the godot 4 extension release?

[Zylann]

There has been a bit of progress, GodotCpp has solved some issues, and the extension can actually run in the editor (#581).
But it still isn't ready for production, as it lacks a lot of testing and crashes might still occur too frequently. The module version remains the most reliable option.

[Shadowblitz16]

@Zylann I would be willing to test the gdextension version with C#

[Zylann]

@Zylann I would be willing to test the gdextension version with C#

You're gonna have a hard time I'm afraid. This is one of the main reasons I'm not fully committing to GDExtension yet, because there is still no easy way for an extension to somehow make its classes available to C# in a native way. You still have to use Get/Set/Call with strings and boxing argument lists and it's just horrible.