[GDExtension] Add fallback configs, and try multiple paths for shared library.

[FailSpy]

This is a draft PR.

General description

The goal of this is to lay some groundwork for more "situation-specific" GDExtension loading. Very basic in implementation, hopefully easy to extend if other types of flexibility is needed.

GDExtensions can now dynamically "fallback" to other configs.

What led me to doing this?

This was built for my own project first, where I had a rather large library I was looking to package independently/optionally, and was hoping to be able to provide "stub" (or reduced-functionality) versions of the same overall GDExtension.

I think this is a reasonable way of doing things, but I'm open to the idea I may have missed something.

Implementation

There are 3 new aspects to the GDExtension file config:

The primary aspect that led to this PR's development:

fallback_extension = {path}

This is a config to a different GDExtension config file. For most cases, I imagine this will be a simple filename (file within same path)

[configuration]
# ...
fallback_extension = 'myextension.stub.gdextension'

If the extension fails to load the shared library attached with the extension, then it will attempt to use the fallback_extension as the config for loading in the "failed" GDExtension object before declaring failure.

If a fallback_extension config fails, and it itself has a defined fallback_extension, then it will repeat.

autoload = {true|false} (defaults to True)

Tells Godot Engine to not auto-load this extension. This was necessary to prevent fallback extensions from being loaded in automatically, and yet still allow it to still be recognized as its own GDExtension resource.
As it seems there's more movement towards building out the GDExtension loader architecture and better supporting things like reloading & hotloading, this seems generally useful as well.

[libraries] section and "possible shared object locations" loading order

Now when specifying a path for a shared object library, you can provide a String as normal, or an array.

First it will choose the tag that best matches as normal, but then will evaluate each path one by one until it finds an existing path. If it does not successfully find a matching file, then it will continue on to autodetection as before.

[libraries]
linux.release = "res://bin/my_extension_linux_release.so" # syntax kept for reverse compatibility and simplicity
linux.debug = "res://bin/my_extension_linux_debug.so"

# If fail to get my_extension_x.dll for these features, then move on to the next until end of list or valid path is found
windows.release = ["res://bin/my_extension_windows_release.dll", "res://bin/alternative_windows_release.dll"]
windows.debug = ["res://bin/my_extension_windows_debug.dll", "res://bin/alternative_windows_debug.dll"]

# similarly for Mac OS, exampling using system paths: 
macos.release = ["/Library/Frameworks/MyExtension.framework/MyExtension.dylib", "/System/Library/Frameworks/AlternativeExtension.framework/AlternativeExtension.dylib"]
macos.debug = ["/Library/Frameworks/MyExtensionDebug.framework/MyExtensionDebug.dylib", "/System/Library/Frameworks/AlternativeExtensionDebug.framework/AlternativeExtensionDebug.dylib"]

This on its own can serve as a lighterweight/easier fallback provider. But this gives less control on the overall GDExtension config per shared library. (e.g. dependencies)

TODO/Uncertainties

I'm still familiarizing myself with Godot's overall architecture/structure, and don't know what I don't know. This will likely want some amount of architectural reworking. I'll update this as I go.

Testing/open questions

• Are paths (absolute, relative, res://, user://) well supported across this PR?
• Does this work well with Mac OS?
• How might fallback extensions go wrong?
• Should a GDExtension resource instead be entirely self-contained, and the idea of "fallback" GDExtensions be pushed to a Project-level setting?
  • This would require building out the idea of "GDExtension" as a Editor-available resource.

Adjacent issues/PRs

#88049 - Adds lookup paths and other path schemes to specifically the GDExtension loading space. This expands the "autodetection" side of things.

[dsnopek]

Can you give more details about the problem this is trying to solve? Why would the library for a particular platform fail to load, yet a fallback would work?

If this is about selecting different libraries for different exports of the same game, you can already do that with feature tags.

For example, let's say you have two Windows exports, one where you want the main library and another that uses the alternative. You can add a custom "alternative_ext" feature tag to one of the export presets, and then in the .gdextension file, do something like:

[libraries]
windows.alternative_ext.release = "res://bin/alternative_windows_release.dll"
windows.release = "res://bin/my_extension_windows_release.dll"

Each line is evaluated in order, so it'll first check if we have the "windows", "alternative_ext" and "release" feature tags, and then if not, move on to the next line.

[dsnopek]

autoload = {true|false} (defaults to True)

I'm not sure we should add something like this. The plan is to allow the developer to enable/disable GDExtensions, probably in Project Settings. And, Project Settings can also have overrides for specific feature tags, so you could use that to control which platforms or export presets a GDExtension is enabled for.

[FailSpy]

@dsnopek So I may be missing something in terms of existing functionality/flexibility for GDExtensions in all of this, but this would all be so a GDExtension developer to deal with scenarios at-runtime on the user-side, not configuring for export by the game developer.

My usecase

I would like a binary that uses a dynamic library if it is available from the user's filesystem. If the user does NOT have that shared library available, there are still certain features I'd like to have available regardless, and I'd like another GDExtension config to take its place.

The intention in my case is to package the full-implementation GDExtension separately from the binary. So, a single binary that can work with the shared lib as-available, INSTEAD OF producing two binaries with different featuresets. Hence why I add "multiple fallbacks" from within a matching feature set.

The full-implementation GDExtension could then be packaged separately for systems that are able to utilize it, and those that aren't don't need to worry about it. Both users have the same binary.

So, to deal with cases where we were unable to successfully load the full shared library from the filesystem, the binary contains a much smaller, simpler "stub" library with a reduced-set of functionality.

How does this solve that?

1. Attempt to load feature-full version of the GDExtension
2. If the library fails to load for any reason (i.e. system incompatibility, missing shared lib of the GDExtension, or lack of a dependency library on the system, etc), fall back to a minimal-dependency and/or feature-reduced version of the library.
3. The "fallback" GDExtension is now only triggered/loaded at runtime if the full one was attempted, and failed to load.

One "GDExtension" overall to me as a gamedev, but the particular underlying implementation is sensitive to what's available on the user's system.

This also allows me to package "extended functionality" as separate to the binary, whilst guaranteeing not losing certain core functionalities of what is effectively the same GDExtension.

Feature tags

I've mentioned why feature tags aren't really the correct implementation for the full scope of the above (desire for a single binary), but it is worth noting that the way its currently implemented, it still doesn't enable fallback paths for finding where a given library is. It simply attempts to find the line where all features are met, and then will fail upon trying to load a missing library.

So this PR also adds the ability to define fallback paths for where to find a shared library from within the committed feature tag line, rather than relying on auto-detection. This is separate to the idea of a fallback extension configuration.

I needed this to introduce a more controlled robust method of finding where my "full-implementation" GDExtension actually is.

Each line is evaluated in order, so it'll first check if we have the "windows", "alternative_ext" and "release" feature tags, and then if not, move on to the next line.

This isn't 100% accurate to what happens, FWIW. Godot, currently, always evaluates all lines and simply commits to the "best match" feature-tags wise and sticks with it. In your particular example config, the ordering doesn't matter.

If alternative_ext is available, then it will always commit to the alternative_ext line for a windows release build regardless of ordering because it has more feature tags than windows.release.

It's only if there are an equal number of matching feature tags that the ordering matters. e.g. windows.feature2.release and windows.alternative_ext.release would be order-dependent in a scenario where BOTH features are available.

godot/core/extension/gdextension_library_loader.cpp

Lines 84 to 93 in 97ef3c8

	bool all_tags_met = true;
	for (int i = 0; i < tags.size(); i++) {
	String tag = tags[i].strip_edges();
	if (!p_has_feature(tag)) {
	all_tags_met = false;
	break;
	}
	}
	if (all_tags_met && tags.size() > best_library_tags.size()) {

autoload config

autoload = {true|false} (defaults to True)

I'm not sure we should add something like this. The plan is to allow the developer to enable/disable GDExtensions, probably in Project Settings. And, Project Settings can also have overrides for specific feature tags, so you could use that to control which platforms or export presets a GDExtension is enabled for.

The purpose I had in implementing this config variable primarily only makes sense if you have duplicate GDExtension configs provided for sake of fallbacks, as a GDExtension developer. You, as the GDExtension developer, need a way to let Godot know that this extension is explicitly not to be auto-loaded.

The developer would still be able to enable/disable a given GDExtension through the eventual Project Settings, which would in turn prevent the fallbacks from loading in as well.
Though it may be worth preventing a "fallback extension" (non-autoloading) from registering in this list to avoid confusing the game developer with the GDExtension dev's intentions.

Right now, any GDExtension available will be recognized by the GDExtension loader and attempt to load in at start.
So, for my use-case above, it's to ensure the stub library only gets loaded as a result of falling back, not by nature of simply being in the Project assets.

We could rename the autoload key to be more explicit about this intention somehow? Perhaps just loads_as_fallback? Not sure.
This config was the simplest solution I could think of for preventing the fallback from loading in as well, as I wanted to minimize .gdextension configs having cross-references or prerequisite conditions to keep auto-loading functionality simple.

[dsnopek]

Ok, I think I understand a little better what you're trying to accomplish.

I'm warming to the idea of an autoload property in the .gdextension. Although, I think it'd be better with a different name, since "autoload" already has a specific meaning in Godot.

We may want to be able to distinguish between "enabled by default" (so, the editor won't automatically enable the extension after it discovers it, but the developer can still enable it to load at startup via project settings) and "manual load only" (so, the developer can't enable it via project settings, it has to be loaded manually via code).

However, the idea of "fallbacks" still seems overly complicated to me. What if the various fallback GDExtensions were all disabled (after we have the ability to have disabled GDExtension), but then you had code that would loop through and use GDExtensionManager::load_extension() to manually load them in order until one succeeded?

[FailSpy]

That implementation is clean and simple, but I struggle to see how a GDExtension developer could configure drop-in fallbacks for the project developer without their involvement, given the chicken-or-egg issue of where the fallback code would reside. It would likely need to be in a GDScript or, if early initialization is required, a third "meta" GDExtension.

I could maybe work around this for my own needs, so I'm open to pulling it from this PR altogether.

On autoload, I think that's very reasonable, so what do you think of a default_load_behavior flag with two possible states: enabled, and deferred.

Until Project Settings supports enabling/disabling GDExtensions, disabled and deferred would essentially function the same, as there’s no way to re-enable a "default disabled" extension at the moment other than by code. So for now, I'm intentionally leaving out a disabled flag. I can add one in with comments if the stub behavior isn't bothersome.

As for fallback paths in the [libraries] section, it might be worth expanding the discussion, as it seems unusual that the .gdextension file defines a static path for the library at all. I see the .gdextension file as more of "GDExtension dev territory," something a game developer shouldn’t have to worry about, yet it still dictates where the library must be placed within the project’s assets.

I think it's still useful to be able to define multiple potential paths, particularly when dealing with loading outside of the Project binary in the user's filesystem, but I do imagine there'll be cases where the project dev will want to override the library path(s) provided by the GDExtension dev, possibly in the future GDExtension-related Project Settings.