Allow extracting asset library assets into subfolders

[aaronfranke]

Describe the project you are working on:

GitHub repo: https://github.com/aaronfranke/GodotExtraMath

AssetLib page: https://godotengine.org/asset-library/asset/408

Describe the problem or limitation you are having in your project:

In my own private projects, I have this repository added as a Git submodule. This allows me to easily update the library in my projects.

The problem or limitation I am facing is that the same directory structure does not work well when published on the asset library. When an asset is downloaded from the asset library from within the Godot editor, it always places everything in the project's root directory.

Describe the feature / enhancement and how it helps to overcome the problem or limitation:

The proposal is to allow specifying a subfolder that the asset should be extracted into. This could either be specified by the user or by the asset creator, though it would probably make more sense to have the asset creator specify the location in case scripts depend on a specific file path.

This means that it would be possible for asset library assets to either be installed through the asset library, or to be installed by adding a Git submodule, without changing the directory structure (something that currently works on the asset library, added as a submodule in the addons folder, would currently have a path like res://addons/my_plugin/addons/my_plugin/plugin.cfg).

Bonus upside: If all the files were extracted into subfolders, then we wouldn't have to worry about README and LICENSE files from different assets all competing to be in the project's root directory (since in a GitHub repository, these files are normally placed in the root directory).

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:

For whatever files are extracted, put them in a subfolder based on a specified plugin_path:

for file in asset_files:
    extract_path = plugin_path + "/" + file_path

Where plugin_path would be something like "addons/my_plugin".

Or maybe we could force assets to go into the addons/ folder:

for file in asset_files:
    extract_path = "addons/" + plugin_name + "/" + file_path

Where plugin_name would be "my_plugin".

If this enhancement will not be used often, can it be worked around with a few lines of script?:

It's not a matter of scripting, it's a matter of convenience and choice, and also a matter of not polluting the root directory with README/etc files.

Is there a reason why this should be core and not an add-on in the asset library?:

Uh... well, these are already add-ons in the asset library 😛

Bugsquad edit (keywords for easier searching): install, installing

[bojidar-bg]

Wouldn't this break all resources within the extracted addon, especially considering the way references are currently saved as absolute res:// paths? (also, consider resources hardcoded in scripts, e.g. load("res://addons/xx/xx.tscn"))

[aaronfranke]

@bojidar-bg The idea is for assets to reference res://addons/xx/xx.tscn etc, but xx.tscn would be stored in the root of the repository, so when the repository is extracted to res://addons/xx/, it will be correct.

There is still a discussion on whether to always use res://addons/something/ or allowing asset authors to specify a custom path.

[Calinou]

To be able to do this, we need to:

Make the API send an asset slug and a normalized version of the asset author's username

This is so the editor knows where the asset should be installed. Users will have to define a slug like-this. (The slug can't be changed by the user once the asset is created.)

If JaneDoe publishes an asset whose slug is like-this, the complete path would be janedoe/like-this. This approach is similar to the one used by Composer and sidesteps most typosquatting issues found in npm.

For existing assets, we can generate the slug automatically by lowercasing the asset name and replacing spaces and other special characters with dashes.

On top of that, the asset slug system will make it possible for third parties to create command line applications to install assets.

Implement recursive detection for editor plugins (done)

This way, plugins can still be loaded if they're scoped on a per-author basis: #1623

---

Since this change expects a whole new folder structure for assets, it's a compatibility-breaking change. This means we can only do it in 4.0.

[Shatur]

Implement recursive detection for editor plugins

Should be easy. I think this might be the starting point. Is someone working on this? If not, then I could send a PR.

[Calinou]

Is someone working on this? If not, then I could send a PR.

As far as I know, no. Go ahead 🙂

[Shatur]

@aaronfranke, could you take a look at godotengine/godot#43734?

[Jummit]

I don't like the idea of being able to install addons everywhere. There is a reason the addons folder exists, it's for organization and clarity.

In my projects I put installed addons into addons/thirdparty, and project-specific addons into addons. In my addon repos the addon files are inside addons, and my package manager installs them either with a hard copy or a symlink to a local git clone, which can easily be updated.

To get to your initial problem:

When an asset is downloaded from the asset library from within the Godot editor, it always places everything in the project's root directory.

This can be solved by having an addons folder inside your plugin repo. The asset library should only download files inside addons.

[Calinou]

The asset library should only download files inside addons.

Unfortunately, doing so would break many existing assets which don't follow this convention.

[Jummit]

The asset library should only download files inside addons.

Unfortunately, doing so would break many existing assets which don't follow this convention.

It would be an easy change: In the download menu only things inside addons should be ticked by default, but you can optionally download everything.

[Calinou]

It would be an easy change: In the download menu only things inside addons should be ticked by default, but you can optionally download everything.

I've thought about doing that, but I'm still worried about breaking user expectations and having to deal with issue reports because dozens of assets will be "broken" out of the box until their author updates them.

[aaronfranke]

@Calinou It can be done in Godot 4.0 and break compatibility. Asset creators will have to update their assets anyway.

[Calinou]

It can be done in Godot 4.0 and break compatibility. Asset creators will have to update their assets anyway.

For 4.0, we'll probably remove the recommendation for add-ons to be stored in the addons/ folder in their repository. Instead, everything should be placed at the root level and the editor will handle placing the downloaded asset to the addons/ folder.

Keeping the current repository structure and checking only the addons/ folder by default could be an acceptable stop-gap solution in 3.x, but it will still break assets unless users manually go out of their way to tick more checkboxes in the install dialog.

[idbrii]

Will forcing addons to install into res://addons prevent them from adding script_templates? (Example asset:
Hierarchical Finite State Machine)

[idbrii]

If I follow correctly, this new pattern means an easier time for users installing from the AssetLib from within Godot. Like before, they cannot change the destination directory so scene references work, but now they shouldn't have conflicts for top-level files.

However, for manual installs it's still possible to make path name mistakes? If an addon is janedoe/like-this, then must be installed to res://addons/janedoe/like-this or scene references like [ext_resource path="res://addons/janedoe/like-this/AddToSceneButton.gd" type="Script" id=1] will break? Would we still want something like the plug:// prefix described in
godotengine/godot#12153 to prevent that?

[atirut-w]

@idbrii if you are asking about my proposed solution then no to both.

Addons will have an optional metadata telling Godot where to extract it to on an opt-in basis as to not break compatibility with 3.x. This way, old addons will still work and new addons won't need the "skeleton" paths.

[idbrii]

I was thinking about Calinou's 4.x plan. If 4.x doesn't allow addons to install outside of the addons folder, then are there other changes necessary (like searching for script_templates inside addons)?

[visuallization]

Wouldn't it be the easiest to have a flat folder structure in the plugin's git repo? So basically no addons/your_plugin folders but all the source files are in root & then you just add the plugin as a submodule to your projects addons directory. Or is the folder structure addons/your_plugin mandatory for godot plugins on the asset lib?

[Zireael07]

Whatever folder structure ends up preferred, it's nigh impossible to enforce it.

[Calinou]

Or is the folder structure addons/your_plugin mandatory for godot plugins on the asset lib?

It's currently mandatory, but this proposal aims to make this kind of packaging redundant for add-ons (all files would be in the repository's top-level and automatically scoped to addons/your_plugin).

[atirut-w]

This could easily be implemented without breaking compat tbh

[Calinou]

This could easily be implemented without breaking compat tbh

Indeed, we can detect the repository structure when installing an asset based on the existence of certain files/folders.

[aaronfranke]

@atirut-w That would be welcome, but it's not a high priority, so it will depend on if a contributor shows up to do it.

[atirut-w]

Why move it to the 5.0 milestone then?

[hsandt]

What about examples? I'm using Yet Another Behavior Tree which has quite useful examples in a separate folder at the root. It'd be nice to still be able to download them. I see a few ways:

a. examples are considered outside the add-on and can optionally be imported at the root. Since examples folder would be besides the addons folder, it is immediately visible in the import hierarchy, you can toggle them depending on whether you need them or not.
b. examples are moved inside addons/[addon name]. User must remember to check/uncheck them as they need. However, since this means we are going to a full import inside the addons/[addon name] folder, the import hierarchy now starts under [addon name], which means the user will immediately see all folders inside it, and is unlikely to miss the examples folder.
c. examples can be downloaded separately by clicking on a different button "Import examples" where user can uncheck examples sub-folders they don't need (similar to Unity package "Download samples..." button)

I think that b. would be the most simple, and as it gets standardized, users will learn to uncheck examples for a lightweight import, or just download them, test them and remove them from disk once they are done.

[hsandt]

OK, this gives us flexibility for Asset Library installs, but how do we tackle install as git submodules? (I come from godotengine/godot#18131 which suggested to work with pure root repo to solve this, but the idea was apparently not retained because it required devs to change their repo structure)

I've tried it again on 4.1, and when a project.godot file is present and you install the repo as submodule, the whole folder is ignored.

@aaronfranke you said you were installing https://github.com/aaronfranke/GodotExtraMath as submodule, how does this work despite the project.godot file? Are C# projects different from GDScript ones?

For now, for submodules, I must either:
a. setup a pure root repository with no addons subfolder (like https://github.com/hsandt/hyper-godot-commons)
b. setup a repo with the official add-on structure (with an addons subfolder) but still no project.godot to avoid disabling folder scanning (which means I cannot test my add-on from a canonical project - not an issue for me as I test my add-ons live in my latest project, but I know many add-ons in the Asset Library have one)
c. setup a repo with the official add-on structure (with an addons subfolder) with a project.godot if I want to, but clone it in some local folder (not as submodule, just as a full repo) then symlink the addons/[addon-name] subfolder (like https://github.com/Jummit/godot-package-manager) - I actually did this for a third-party add-on I wanted to contribute to

It seems that only #1205 is still tackling this issue: it suggests full support of sub-projects, which means project.godot files wouldn't be an issue anymore and we'd be able to scan such add-on folders installed as git submodule (although in practice a given project will only care about what happens inside its own addons subfolder).

Still not the ideal addons/[addon-name] unified structure as we'd have Asset Library installs under addons/[addon-name] vs git submodule installs under [git-submodule-repo]/addons/[addon-name] but acceptable (it's not a bad idea to distinguish add-ons we know we can edit vs read-only anyway). I suppose I'll just follow #1205 from here, but I wanted to stress this because the custom addon repo/submodule talk kinda got lost in the process.

[aaronfranke]

@hsandt I changed the structure of that repo to fit in better with Godot's asset library.

[tjpotts]

setup a repo with the official add-on structure (with an addons subfolder) with a project.godot if I want to, but clone it in some local folder (not as submodule, just as a full repo) then symlink the addons/[addon-name] subfolder (like https://github.com/Jummit/godot-package-manager) - I actually did this for a third-party add-on I wanted to contribute to

For the benefit of anyone else that arrives here trying to do this with a submodule for a C# addon: you may get errors if the git submodule is within your godot project directory, even if there is a .gdignore file preventing Godot from seeing the submodule. I was able to get it working by manually adding the following to my .csproj, where external is the directory containing my git submodules:

    <ItemGroup>
      <Content Remove="external\**" />
      <Compile Remove="external\**" />
      <EmbeddedResource Remove="external\**" />
      <None Remove="external\**" />
    </ItemGroup>

I'm a C# newb, but I assume the C# project is simply building everything in the project directory with a .cs extension (the errors were referencing the external/ directory, and complaining about duplicate definitions). If that's the case, I think automatically updating the .csproj to exclude .gdignored directories would be a nice feature.

[whogben]

Hey guys, bumping this thread because it seems to be the most relevant one for the issue.

As a summary:

• The AssetLib always installs addons to the root of the project, not into the addons folder
• Using a GitHub repo to host an addon requires putting the addon in /addons/your_addon, while you have to put the README.md and LICENSE in the root of the repo
• This creates conflicts with other addons, installing multiple addons will attempt to put multiple competing copies of README.md and LICENSE in the project
• This also prevents you from developing multiple addons in one project, as they'd have competing Git repos all vying to control the root project directory.

Workarounds:

• redundant copies of LICENSE and README - make sure to keep a redundant copy of these files inside the /addons/your_addon directory, keep them up to date with the ones at the root of the dir, and leave end users the problem of deleting the overlapping LICENSE and README from various addons they install
• ??? any other workarounds?

Fix Suggestion:

• Add an option to the assetlibrary submission process "Installs into /addons" which can be checked by the addon submitter, and causes Godot to install the addon's root into '/addons'

If anyone is aware of a workaround that would allow me to have multiple addons in the same project please let me know. Right now, I have several addons that depend on each other. I would like to have them in the same project but the README.md and LICENSE would conflict, as well as the git repos for each addon fighting over the root project directory. For now I will look for a way to unpublish the addons and if anyone wants them, they can get them from github and manually extract into /addons.