Rework how Godot handles built-in scripts

[SysError99]

Describe the project you are working on

I have one of my project that may take advantage of this feature, but currently not needed.

Describe the problem or limitation you are having in your project

My personal thought is that built-in scripts should be default to not exposing them in the file system and keep whole project tree organized & clean. I believe more than 90% of number of scripts don't even have any dependencies or cross-file references. The problem is, GDScript isn't designed to do this, and even worse is that VCS is broken with it. Still, many of bugs can be fixed, and we all know that this feature is very useful, also very useless in its own ways, we can still rework how Godot stores scripts in TSCN files.

[sub_resource type="GDScript" id=1]
script/source = "extends Sprite

func _ready():
	print(\"Hello\")
"

I personally find this to be unfavorable way to store scripts inside the file TSCN files. Even though it makes parsing implementations much more easier, but as I (and many people) have said, it breaks VCS horribly.

I tried thinking about best way to store scripts, and read some of possible ways to handle built-in scripts. (#1909, and #1274) Which both are better than current method Godot is using (at least, in my opinion). Still, both also has some of limitations:

1. On "discrete scripts", it lacks of portability, which means it can't be easily moved and be used in another work or project without any of discrete menu or manual copy-paste. And definitely be a problem if you manage files outside of Godot.
2. On SceneScript, which I'd prefer the most. The main problem is another abstraction of Script` with some of extensions that can be favorable, but reworking into support it isn't that easy.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Since Godot is still focused on ease of use, built-in scripts are still very useful and powerful for many of people who aren't specialized in programming and organizing stuffs.

The biggest advantage of my method is, hopefully, my implementation idea can be easy to apply, solve issues of current and both of methods I described above, and practical in real-world usages.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I stepped on my all-time favorite game "Stranded II", a survival sand-box game. I remembered the day I modify this game and its scripts, that look something like this:

# ITS PROPERTIES
loot=4,1
loot=23,1
maxweight=12341
script=start
	on:create {
		timer "self",3000,0;
	}
	on:start {
		timer "self",3000,0;
	}
	on:timer {
			
	}
	on:use {
		if (health("self")>0){
			exchange "self";
		}
	}
	on:kill {
		freetimers "self";
		event "iskill_hunt","global";
	}
script=end

Usually, the file that store these lines are used for storing many of object properties that can be modified, including its behaviors via scripts. The layout is easy and clean to read, even with common Notepad program. I find this to be quite efficient method and not too difficult to implement into Godot.

The way the file can achieve multiple objects & scripts in single file is to define each object's ID, and assign to the object

id=1
script=start
...SCRIPT...
script=end

id=2
script=start
...ANOTHER_SCRIPT...
script=end

In Godot, we can achieve same thing. This is some of example of how it would look:

<script type="GDScript" id=1>
extends Node
    
    func _ready():
        print("Hello from built-in script 1!")
</script>
<script type="GDScript" id=2>
extends Node
    
    func _ready():
        print("Hello from built-in script 2!")
</script>

There are two ways I can think of to implement this into Godot.

1. Embed in TSCN as usual in a custom block or at last line of file. This is the best way if we still prefer integrating everything in TSCN itself, but may not be an option if interfering with TSCN parser can break the engine or needs a lot of code rewrite.

[gd_scene load_steps=6 format=2]

<script type="GDScript" id=1> // This has different character so it shouldn't be any problem with parser.
extends Node
    
    func _ready():
        print("Hello from built-in script 1!")
</script>
<script type="GDScript" id=2>
extends Node
    
    func _ready():
        print("Hello from built-in script 2!")
</script>

2. Separate built-in scripts in a custom file. This add one more file to be carried along. The main TSCN file can be bound with this type of file using built-in script tag.

If this enhancement will not be used often, can it be worked around with a few lines of script?

This can get around very easily with common directory-style to store all node scripts. Still, this is unfavorable for many people who only write only few lines of script for nodes.

- level
  - level_1.src
    - super_easy_trapdoor.gd
    - i_just_wanna_say_hello.gd
  - level_1.tscn
  - level_1.gd

Is there a reason why this should be core and not an add-on in the asset library?

This change behavior of the engine directly, and needs rewrite certain parts of the engine.

[KoBeWi]

So you basically suggest that built-in scripts have to be stored in a different way than currently? But still inside scenes? How is that different from the current system?

[SysError99]

The main benefit is to make a better support with VCS, make it more friendly to script parser, and also make better support of type linting for external editors (especially when making it as separate file instead of integrating it into TSCN files)

This should not be much of difference compared to the old system but to make a better method to store built-in scripts than putting them in string format, which does not serve the purpose quite well. Still, considering Godot isn't designed for built-in scripts, this method can be (quite) easily appended into creating temporary GD files that are hidden to the user, silently binds to the scene, and synchronize changes with built-in scripts. This way we can (at least, partially) solve few issues (such as the use of class_name, Find in Files will also work since it will also search for hidden GD files, and tool should also work)

[dalexeev]

#1274 (comment)

And the problem with escaping double quotes and backslashes can be fixed by adding a property_hint that tells Godot to save the string like this:

[gd_scene load_steps=2 format=2]

[sub_resource type="GDScript" id=1]
script/source = <<<EOL
extends Node

func _ready():
	print("Hello!")
EOL>>>

[node name="Node" type="Node"]
script = SubResource( 1 )

[YuriSizov]

This should not be much of difference compared to the old system but to make a better method to store built-in scripts than putting them in string format

You probably meant something else here, because scripts are by their nature text, i.e. they are in a string format from the get-go. The change from normal Godot serialization to some XML-like tags doesn't make much of a difference for any tools.

A built-in script is always going to have issues with third-party tools because they'd need to parse the whole file containing the script. Maybe you can get around it by always putting such properties at the end and having a header field that informs the parser where the script content starts in bytes, or characters, so it can skip a bunch of it. This will make support for extended features in Godot itself more feasible, such as class_name, searching, and tool mode. And it can help with VCS a bit, but it won't make support for external editors any different, they still would be required to introduce some way of parsing Godot-specific files.

And storing built-in scripts in separate files makes them... not built-in.

---

If you are okay with having more files, but would still like to lower the cognitive load from looking at the file system dock, many IDEs have a feature that collapses different related files into one entry that you can expand (say, in the world of web dev, a minified script can be visually collapsed with its expanded form in the project panel). This may be a solution that suits you.

[KoBeWi]

The main benefit is to make a better support with VCS

But they are already stored as text, so they are VCS-friendly. The only problem is that two people can't work on the same scene when you edit its built-in script, but editing scripts without touching the scene is not common anyways.

make it more friendly to script parser

It makes no difference to the script parser. The Script resource is extracted from the scene before it's being parsed.

a better method to store built-in scripts than putting them in string format

Technically, all scripts are strings 🙃 Except VisualScript.

considering Godot isn't designed for built-in scripts

This is not true. Built-in scripts exist specifically because of how Godot can handle internal resources. They are just a special case of that.

Find in Files will also work since it will also search for hidden GD files

You can add tscn to the searched extensions and built-in scripts will be included too. The only problem is that you can't open the script directly (only the scene) and you need to mind the escaped quotes when searching for strings.

tool should also work

It already works. There were problems with reloading built-in tool scripts, but it's fixed in Godot 4.0 and 3.5.

[dalexeev]

A built-in script is always going to have issues with third-party tools because they'd need to parse the whole file containing the script. Maybe you can get around it by always putting such properties at the end and having a header field that informs the parser where the script content starts in bytes, or characters, so it can skip a bunch of it.

When working with scenes from the Godot editor, we can use temporary files for external code editors, and these editors may not even know anything about the TSCN format.

Built-in scripts are similar to inline scripts in HTML: while this is a bad approach for large scripts, it's very convenient for small scripts because it does not clutter up the file system. I think both types of scripts are needed as both have different advantages and disadvantages. But I'm still against intermediate options like "discrete scripts".

This proposal aims to improve the built-in scripts and make them even more VCS-friendly, since removes the escaping of quotes and backslashes. The way we can achieve this might be a little different (see my comment above), but there is nothing wrong with this proposal: external scripts remain external, and built-in scripts remain built-in. Removing built-in scripts is not part of this proposal.