-
-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement custom resource path support #6307
Implement custom resource path support #6307
Comments
On my RPG framework StarEngine, I implemented a custom scripting language, StarScript, which I wanted to be able to quickly refer to resources without having to load them as export variables. Paths are clunky and can change, so I created a class Then the resources can be loaded from StarScript with Would this system if implemented make this addon redundant? And if a custom path root is under |
@reduz Some thoughts:
Maybe the overlap I'm seeing between resource packs and resource file systems isn't correct. But as they both are basically just a source of resource files, the two should not end up as different, incompatible systems. Ideas for the future:
@mrpedrobraga No need to store the loaded reference, since Godot already caches resources by default. Multiple |
I had suggested this kind of Custom Resource Path support in the discussions. I had kind of tinkered with the idea of custom resource paths which does the save, load and overwrite operations. |
+1 for mounting points specifically. It seems like a good way to swap a large amount of resource pointers without loading a whole I'd imagine calling |
Is it the right thing to do here to use the "protocol" of the path for what I could consider "namespaces"? My only concern here is how easy it is to get 2 things to clash, and how much work is needed on the user (here the developer using the addon/library) to prevent those, and how brittle the code may become from it? We need this idea, the only thing I'll question is if this exact way is the right one. For example what happens if a 2 packs I pull in declare 2 file accesses with the same name? |
I can imagine this working with something like Steam Cloud. I could easily read/write files using the |
What I was thinking is to map the resources to a rest api/s3 bucket and load assets from there. |
I love the idea but is there a way to set the custom paths in ProjectSettings? I'd like them to be available at startup so we can use them in ProjectSettings such as Log Path. I always wanted to output my logs to a folder next to the executable and I can't find a way to do it currently. |
Or in EditorSettings For global plugins imo it would be better to use editor settings folder for editor plugins data |
Just an observation. I think it's better to avoid using Godot's existing It would have been better if it had been |
I don't think the existing system has ever meant to imply it follows the URI specification. The I agree that if it were |
Save this for Godot 5? I don't know when/if that would happen, but it may be a good thing to remember.
Yes, now it was done like this, it has to match. It being normal URI spec would indeed be better, but if we get it in Godot 4 then it has to follow this spec. |
@Frontrider Godot 5 is years down the line. |
I know, mainly just noted that it is that territory. |
Agree that this seems like a major change, and to keep a "proper URI compatibility" for On the other hand, I'm pretty sure In any case, I was concerned because of the examples given for the API; it should not include slashes: FileAccess.add_resource_path("game","/home/user/Documents/somepath")
# note "game" instead of "game://"; the :// or whatever amount of slashes should be implied Still, to be honest, for custom resource paths that I define myself, I would expect a well-defined and well-known standard of URI compatibility, so the examples above would make much more sense like this:
Not just for the slashes/authority part, but the whole thing, including query parameters and fragments, all of those matching URI could be useful for users' arbitrary definitions. |
The res:/// format feels like a workaround. Indeed, the amount of code changes and the impact on existing code may be minimal, but there is no doubt that it will become a technical debt in the future. I feel it would be more beneficial to adhere to a format that can satisfy the URI specifications. |
I believe the core of this discussion centers on how to avoid collisions within the scheme. Firstly, there are three combinations of problems: Engine vs Engine: Engine vs Plugin:
By enforcing such prefixes, it’s possible to separate engine functions from user extensions, which should prevent any hindrance to future engine development. Plugin vs Plugin: I consider this issue a high priority as the current res:// format is not suited for large-scale development. |
This is actually good, but since Godot 4 is out the way it is now it should not be changed/broken. For Godot 5, make it happen. |
See also:
This is an interesting idea. But we should think about making sure that custom protocols don't break/shadow engine or other commonly used protocols. In the code base I found mentions of the following protocols:
|
Perhaps it would be a good idea to have the engine and plugins define up-front all of the different protocols that they use? For example, via a read-only ProjectSettings property and the Back when I had more time, I was working on something akin to bitbrain's Pandora project (a Resource-driven "RPG database"), but ran into problems in early days: I wanted a centralized pipeline for processing over whatever users were saving into it w/o resorting to a separate singleton (just good 'ol |
Another relevant comment from a different proposal that suggests inclusion of a |
I think the things discussed here are sounding increasingly similar to a Virtual File System. I'm currently using one (https://github.com/xoofx/zio/) in my project so that we can reference external files using virtual paths (like As for the protocols, What's even better on using VFSes is that every VFS implementation is a working file system on its own, so you can imagine use cases like giving different VFS to different plugins or subprojects, so that every plugin and loaded package can work on their favorite file system mounting scheme, without worrying about breaking other parts of the program. You can give every pack their own mounted VFS, so inter-pack and intra-pack references are nothing more than absolute and relative paths, you get to control which folders can be seen from where, and additionally we get free sandboxing for packs. VFS isn't without problems. Aside from needing to rewrite a large portion of file managing code, call paths might become significantly longer with many virtual calls in them, and performance might degrade when too many VFS layers are overlaid together. You might also lose a global file system shared by everyone if sandboxing becomes a thing. It's just my two cents, and please take it with some salt. Disclaimer: I'm still new to Godot (I fled from Unity in August), and haven't looked into Godot's code yet. All my knowledge on how Godot's file system is implemented comes from skimming through issues like this one. I may be wrong in some places, and if that's the case, I would appreciate a kind correction. Footnotes
|
That is the better version it. That also solves the problem of "what if the devs need local assets, while the finished product uses remote ones". You can control it from 1 location in the code. |
The whole |
https://github.com/godotengine/godot/blob/6543495b49613d20f7e32f2b9d38e4a2f1d06db1/core/io/file_access.h#L45-L46 FileAccess is already a virtual file system and especially if it's FileAccessMemory or a FileAccessPack. |
Then we have a second reason for why it might be a good idea to do it through that. I like that idea for one reason. You could mount dev resources from res or an external folder/pck, while in production you can mount say web resources without touching anything in your code except 1 place where you toggle it. Instead of having 1 if for each resource you need to access. |
Yeah, I'm more from a current state perspective, that Custom URI protocols might be beneficial compared to pure paths on things that aren't naturally directory-structured. For example, things like
Then we have an even better reason for that! In this case, it won't be that much work (compared to what I expected) to write a Footnotes |
FileAccess is only kind of a VFS specifically because res and user are hardcoded into it, (and even then its not really FileAccess that's doing that) if resource paths that aren't local assets are included, it would cease being one, unless you limit FileAccess from doing that but then it would disagree with the rest of the interface and would be incapable for clearly forbidding certain types of access behaviors, like FileAccess would have no idea if its trying to look at a URL with a VFS without checking the whole path. VFS is just honestly way more limiting then relying on protocols, I find it way more useful to actually see and indicate a clear accessing something without trying to traverse some path. |
I think the question between "scheme" and "vfs" is what this is for.
|
I think it's important for us now to clarify what a resource path is designed for, and more importantly, what it is not designed for, and then talk about the redesign accordingly. As far as I'm concerned, it's currently used in two places:
If we want to redesign this system, whether by implementing new protocols, or by doing a complete redesign ground-up, we must support the two use cases above somehow, and try not to deviate too far from it. Resource paths are, after all, strings, and should not be abused to replace function calls. Things that, in my opinion, could be supported after extending resource paths (regardless of how):
Things that, in my opinion, should not be supported even after we extend resource paths:
Finally, I would like to hear about other participants' ideas on what we should and shouldn't do in resource paths, especially from experienced Godot developers. |
This one did get me to think, but I do not think this is going to be a use case in the future. We know that c# will become GDExtension so you will not reference the c# class through the resource loader, but via the class registry. Secondly, what you need for this is the ability to pass down an arbitrary string to a handler that returns the resource. Which in turn opens up the resource loader to handle anything people can plug into it. Could be useful, but I'd rather not go there.
I only see the VFS as a solution for these two, because both are basically just file paths. They map seamlessly to that. But external files do have issues when you try to pull them in, and those may apply to remote files as well. May be more feasible to have this piggyback on the PCK loading. |
Accessing a remote resource really does not work for a VFS, a VFS doesn't define any expectations for its interface (and specifically dealing with a remote resource absolutely needs user awareness) and they're not really ever used for that that (and I'm not just referring to local remotes like a local network) and I most certainly wouldn't want to use it for that purpose, a protocol gives clear expectations, or in the least can designate clearer expectations to the user, it distinguishes itself from a generic filesystem which it may never act like or may even outright violate. |
If that's what's going to happen, I would then recommend restricting the use case for resource paths I mentioned above to just local path-like structures (directories and compressed directories). |
Godot itself has a system/expectation for it, namely the load/preload methods. It is already expected that Godot has to properly load in larger resources when you call |
Maybe we could separate |
Again, it's detrimental to have something that looks like a URI but isn't. This should change, at least in Godot 5.
Also put all godot-specific path formats under it. For Godot 4 compatibility, map from the current format in res/user/etc. paths to a canonical godot URI, so either form is accepted when reading, but URIs are used when writing.
|
Instead of a complex solution "one day in the future" I would love to see just a simple way to override As far as I understood, they are defined in os_xxx.ccp like this:
If I was able to inject my own implementation of FileAccessWindows & DirAccessWindows in godot-cpp extension, I would be able to implement my own "virtual fs" and "custom protocol" without any changes required in godot code. |
The problem with such an implementation to me seems like your special FileAcess override would not be composable with other "virtual fs" plugins. To support all desired paths it would be needed to implement all of them in the fs. This is not scalable for the simple and common usecase of package management. Most languages that are used with godot that are not gdscript have a package manager of some kind whether it be official or third-party. It seems like a common usecase to me for it to be possible to load gdextensions as packages that would otherwise need to be vendored into the project from such package managers (eg Now if language integration addons/modules want to use a virtual fs api what happens when a common usecase like the ones mentioned at the top appear? Every language integration with a package manager that wishes to use the fs to load packages whether it be for scripting or gdextension would need to reimplement all expected functionality from all existing addons/modules to get feature parity. To me this does not seem like a "one day in the future problem" to me. It seems like a here and now problem as someone who has the problem that I am adverse to vendored dependencies to the point I'd rather create a package manager for godot or else adapt an existing one just so I can depend on useful addons without polluting my git tree |
Describe the project you are working on
Godot
Describe the problem or limitation you are having in your project
There are several cases where the resource file paths offered by Godot are not flexible enough, and community has complained about this.
Godot currently hardcodes
res://
anduser://
, but users often require more flexibility here.Describe the feature / enhancement and how it helps to overcome the problem or limitation
This feature would allow to add custom resource path providers to the filesystem. Users would need to pass a class and some configuration information. Probably an API like this:
This would allow, as an example, to have global editor plugins. As the editor could register an
editor://
path where the global plugins are saved.Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams
Currently the
res://
anduser://
paths are hardcoded, and a lot of places in the codebase check for "res://' manualy to validate resource files. This would need to be cleaned up to look up into a FileAccess function.Implementing this is not specifically hard, since what determines the base path and the File/DirAccess to use is mostly on
open()
.If this enhancement will not be used often, can it be worked around with a few lines of script?
N/A
Is there a reason why this should be core and not an add-on in the asset library?
N/A
The text was updated successfully, but these errors were encountered: