Add a FileSystemProtocol to handle the path prefixes instead of doing it hard-coded in FileAccess

[TML233]

Describe the project you are working on

A game with mod support.

Describe the problem or limitation you are having in your project

You can't easily add a file path prefix to the cpp engine code. You have to deal with these hardcoded prefix checks in FileAccess.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

This is another design which greatly enhanced the flexibility of path prefixes than the one described in #6307

Protocol is the prefix part of a file path. For example, res:// user://.

FileSystem singleton manages FileSystemProtocols and provides a central entry point for methods like open_file create_directory. These methods parse the path given and extract the protocol part to determine which FileSystemProtocol to use. The method calls are then delegated to FileSystemProtocols.

FileSystemProtocol is a base class which handles how each protocol access files. It can use different FileAccess classes for handling different file sources. It can also just delegate the work to other Protocols. For example, a user custom Protocol mods:// can remap the paths to res://mods/. We can also expose FileSystemProtocol to the script to allow users write complex protocol behaviors.

FileAccess access type is removed, because they are handled in FileSystem redirecting. FileAccess of a specific OS platform only access the path of the OS filesystem (simply say, the disk, or the network path handled by os). Other protocols can delegate the work of loading files on disk to the os:// protocol. By using the newly added FileAccess::set_path_disguise, other protocols can make the FileAccesses returned by the os:// protocol disguise as they were from a path with different protocol prefix.

An example of what the work flow is
Let's say you are calling FileSystem::open_file("res://icon.svg", FileAccess::READ, &err). FileSystem recognize that we are using the res:// protocol, so it calls FileSystemProtocol::open_file("res://icon.svg", FileAccess::READ, &err) of the registered FileSystemProtocolResources. The protocol first checks packed resources, and the file isn't there. It then transforms the res:// path to the globalized path of os filesystem and calls the open_file method on FileSystemProtocolOS. If the OS protocol found the file, a FileAccess is returned to the res protocol. The res protocol then returns the file to the FileSystem. FileSystem then ask the res protocol whether to disguise the file path (this is separated from open_file because multiple protocol names can point towards the same protocol object, this ask-disguise step provides the current protocol name used to the protocol object). The res protocol then calls FIleAccess::set_path_disguise(original_res_path) to make the file from the os to look like it was from the res:// path.

Future proposal for the res:// protocol
We can manage a resource mounting stack in the res:// protocol, and it allows you mount different resource sources like a pck file, a zip file or a path on the disk into the res:// virtual path. Archived sources like pck and zip work just like how they currently work: they load their file table when they mount, and load the actual file when a specific file is loaded.

When a file is requested, the file is searched in source stack from the most prioritized to the lest prioritized. This ensures that every mounted source can be unmounted easily, and files can be found properly when the mounting stack is reordered (players can change the mod load order on the fly). Each resource source has can_write property. When a write access is requested, the stack is searched for the first source which can be written to.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

class_name FileSystemProtocolCustom
extends FIleSystemProtocol

# We might delegate our work to the os
var protocol_os: FileSystemProtocol;

func _init():
    protocol_os = FIleSystem.get_protocol("os")

func _globalize_path(path: String) -> String:
    # Transform the protocol path to the os path.
    return "C:/"+path;

func _open_file(path: String, mode: int) -> FileAccess:
    # Handle the open file request in our own way...
    # ...

    # We might need to delegate the open request to the os
    var os_path = globalize_path(path)
    var file = protocol_os.open_file(path, mode)

    return file

# FileSystem got the opened file, and it now asks us whether to disguise the file path.
func _disguise_file(file: FileAccess, protocol_name: String, path: String):
    # For most of the protocols, they want their files be de disguised under their own prefix.
    # But this is not for the os protocol, so the os protocol can just don't disguise file path here.
    file.set_path_disguise(protocol_name+"://"+path)

# In some place we can add our protocol to the FileSystem
var protocol_custom = FileSystemProtocolCustom.new()
# Adds custom://
FileSystem.add_protocol("custom", protocol_custom)
# Adds test:// pointing to the same protocol object
FileSystem.add_protocol("test", protocol_custom)

If this enhancement will not be used often, can it be worked around with a few lines of script?

It can, but this proposal regulates how path prefixes are handled. It makes future code maintainence easier, since more and more file prefixes are adding in, like mem:// and csharp://.

Is there a reason why this should be core and not an add-on in the asset library?

This proposal regulates how path prefixes are handled. It makes future code maintainence easier, since more and more file prefixes are adding in, like mem:// and csharp://.
This proposal also is a prerequisite for the resource mounting stack described above.

[HolonProduction]

Absolutely no opinion on this, but before merging any implementation, there should go some thought into whether we need to reserve certain protocols for future use. Specifically thinking about global plugins.

[HolonProduction]

Also personally I'd reserve mods for the time being. Making it usable from the get go, would lead to different modding systems using the same protocol which isn't great for the eco system IMO. Modding implementations should stick to something more unique. If a standard stabilizes and maybe even becomes cross compatible (think like package.json powering npm, yarn, deno,...) that would be the time to introduce a mods protocol.

[TML233]

Also personally I'd reserve mods for the time being. Making it usable from the get go, would lead to different modding systems using the same protocol which isn't great for the eco system IMO. Modding implementations should stick to something more unique. If a standard stabilizes and maybe even becomes cross compatible (think like package.json powering npm, yarn, deno,...) that would be the time to introduce a mods protocol.

I think we should first tackle the problem of: mounting multiple sources of resources path under different paths instead of always under the root of res://.
Also the path used in the mod pack should be in a relative manner to allow the base game mount the mod under other directories or protocols. We might need to modify the res loader to allow loading external resources whose paths are relative. And a button that toggles the current path scheme (res:// or relative) should be added in the editor too.

After solving those problems, reserving the mod:// protocol won't be that important, since you can basically mount the mod pack anywhere you like, and the path references inside the mod pack are resolved properly.