New Godot workflow solution: create new Object types instead of attaching Scripts

[samdze]

This issue represents a whole new solution on how Godot workflow might work.

Keywords here are: semplicity, consistency, solidity, reliability, intuitiveness, naturalness, integration.
Changes to the core engine are minimal. Some changes to the editor.

The general idea is explained here:

TL;DR; Replace the current "attach a script to an existing object" user approach with a more solid, simple, reliable, coherent and powerful "create new objects extending existing ones" approach to obtain custom behaviours.

• Plus: much better support for plugins, custom nodes, custom resources. Cross-language support.
• Pro: single design solution that fulfills coherently every high end need of flexibility and customization and that increases semplicity.
• Con: to be able to provide a globally coherent, intuitive and simple system, some breaking changes are needed.

1st part, workflow, here's a more in-depth explanation:

In my opinion, Godot needs an easy and quick way of defining custom types that act like built-in ones throughout the project.
I used and created several game engines, and I have to say that I really like the core design of Godot, based on different types of nodes that are composed and combined to form trees structures.
It makes very clear and immediate what is the purpose of every component of the game.

However: I'll say it directly, the entire concept of obtaining a custom behaviour only through a script attached to an existing engine node is not solid.
To me, that is not what a full-featured, powerful and serious engine should be able to provide and decreases drastically the ease with which new features would be added.
There's need for a more powerful, reliable and unique way to describe gameplay, tools and plugins logic.

The fact is, Godot has a superfluous concept: scripts that must be attached to a node;
To do everything useful the engine is capable to do right now, Godot doesn't need an extra concept like that besides the core design solution: the Node.
Nodes themselves can provide everything needed following the natural hierarchy that Godot defines. Scripts intended as objects that have to be attached to a Node are just one more completely avoidable extra complexity.

The issue is that Godot uses scripts (composition) to mimic the behaviour of inheritance.
Doing that through "scripts that have to be attached to an object" only uselessly adds an extra layer of complexity to the user because no more than one script can be attached to an object: making in fact the extra complexity of having to handle composition completelely useless and worthless.
Godot has a wonderful composition system based on trees of nodes, let's leave it there without touching nodes themselves.

How:

The solution is to encapsulate the "script" withing the Object class and to discard completely this "attach script to a node" user approach, making the user only be able to define new nodes (or in general, new Objects) inheriting existing engine-ones. (to the user eyes)
From now on, I will refer to the concept of a script to attach to a node as "Godot Script", and I will refer to the new type of script that defines new node types as "Custom Node Script".

Every use case of the Godot Script will be covered, but this approach capabilities go beyond that:

1. Types of nodes and types of scripts are no more a different thing, since nodes can no longer have a Godot Script attached, their behaviour is defined only by their type, increasing semplicity, clearness and highlighting the real tree-of-nodes Godot core design.
2. Custom Node Scripts can be used trasparently like built-in nodes with the convenience of having them shown in the "Create New Node" window automatically.
3. Plugins could easily define their own Custom Node Scripts and let them available only if the plugin is active.
4. Custom engine-like nodes have not to be registered "statically" via the add_custom_node method of EditorPlugin, every Custom Node Script created in the project is automatically added as a custom engine-like node.
   This is particularly useful when custom engine-like nodes could have a variable number of subclasses not known a priori by the plugin developer, for example.
5. The editor UI can be left almost as it is. The script icon to the right of a node indicates that is an instance of a Custom Node Script, and clicking it will bring to the script editor of that Custon Node Script.
   Only difference is: that "script" cannot be removed, because to the user eyes that is the type definition of the node, much like you cannot remove the KinematicBody-ness from a KinematicBody.
   If the user wants to remove/change the behaviour of a node, he'll have to change its type. The UI could support the quick creation of a Custom Node Script extending the selected one. (Like how it works now with Godot Scripts)

No more "Attach Script" and "Clear Script" on right click on a node in the SceneTree dock (nor next to the "filter nodes" bar, maybe to be changed with a Change Type button). The "Change Type" option will be the only necessary one.
Creation of new Custom Node Scripts could be provided adding an option to "Create New Node" and "Change Type" windows, allowing the user to create a new node extending the selected one providing file name and file path.

The only use-case not supported is the current "attach a script extending a node higher in the class hierarchy than the node it is attached to", however that's a very limited and not so useful use-case.

Now, talking about how to automatically generate custom engine-like nodes from Custom Node Script files:

1. A Custom Node Script is saved in the file system. (GDScript, C# or whatever)
2. Godot scans the res:// folder and detects the Custom Node Script file.
3. Godot generates a default .import file for that Custom Node Script, extrapolates superclass info, and defines a default icon (default icons for each letter of the alphabet or a set of utilizable pre-made icons would be great), description, whether the node should be displayed in the "Create New Node" window (maybe off by default) etc.
   Data only relevant to the editor and little more.
4. The user can then open the Import dock to customize the generated .import file for the selected Custom Node Script.
5. Godot retrieves all the .import files to be able to correctly show custom types in the "Create New Node" window and to let the user use identifiers instead of filepaths.

The same solution also applies for Resources, they should appear and be created in the "Create New Resource" window and act totally like built-in resources to the user. (However, making clear that they are custom)
Custom Nodes and Custom Resource names/identifiers should be usable like built-in types identifiers. (export variables, for example)

Considerations:

This system will make the creation of custom reusable nodes immediate for games, tools, plugins, etc.
No need to simulate any corner case because the system itself would be specifically made to only support new engine-like Objects creation.
The changes to the core engine itself would be very little, and if I were to develop the best possible engine, I would absolutely prefer this way over the "attach script to an object" one.

Support for the creation of custom engine-like nodes that are in fact scenes is natural and intuitive.
Scenes created with a built-in root Custom Node Script would automatically be processed like Custom Node Script files by the import dock.
Having the Custom Node Script built-in into the scene makes sure the Script (Node Type) is owned only by that specific scene, making Node Type -> Scene a unique correspondence.
Such scenes become accessible and instantiable through filepath, through "Create New Node" window and through (custom node) identifier.
This would be a very powerful feature, perfectly integrated into the new system.

One can still argue: "What changes or how it is better than the current system? This, that and the other things can be done in the current system too, so what's the point?" and that's probably correct.
But here we are not talking about new things that can be done. We are talking about the convenience, the speed, the intuitiveness, the elegance, the solidity, the reliability, the consistency of how things are done.
This is the whole point of a game engine. What can you do with a pre-made game engine that you can't do without? Nothing, you can write it yourself or develop a game without a clear disjunction between engine and game.
It is all about convenience, speed, the power of doing the right things in the minimum amount of time. A game engine hides complexity and imposes constraints, but they usually are the RIGHT constraints, the constraints that permit a project to be built in a solid and quick manner.
Ideally, constraints that enforce a structure that allows everything that should be doable, doable in a intuitive, quick and powerful way, and everything that shouldn't be doable, denied or redirected towards the right way that ensures solidity, productivity and organization.
This is what this new system is all about.
This system permits everything the old one does, but permits it in a way that offers much more solidity and convenience, also semplifying the user-level process.
There are fewer components the user has to cooperate with, but the remaining ones allow him to do everything he was able to in a more intuitive way.

Note: the solutions just explained would become much less solid if the old Godot Script was mantained. Users could change the instanced scene root script (via editor or at runtime) and get a broken custom node, toghether with other similiar possible situations that are just not desiderable.
Similar problems are valid in many other areas of the engine: everything that "composition simulating inheritance" offers more than simple inheritance is the possibility given to the user to make mistakes, only that.

The Custom Node Script approach provides just the right amount of control to the user to let him do whatever he wants reliably.

2nd part, cross-language support:

Regarding the implementation of this system in a cross-language manner:
Every language module should provide a common interface to the core engine, which provides an interface itself, letting them define custom types through a unique entry point.
Custom Node Scripts created with language A should easily refer to Custom Node Scripts created with language B and viceversa.
There must not be the eventuality that some plugins remain unusable within a project that uses a different programming language.
This is important to promote the creation of globally reusable plugins by the community: an engine that is hard/tedious to extend and to customize will never reach high levels.

Possible solution to interface different languages and make them communicate:

1. The engine core defines the requirements and the limitations that every language module should take care of and provides them the interface to register and get informations on types.
   That means that there must be a "basic" set of language features that every language module has to offer. Features exceeding that basic set of features will not be become cross-language.
   (Example: C# Generics will not be usable/visible in GDScript, only in C#). This means that plugins implemented in language "A" can internally use language-specific features, but will only expose globally accepted features to other languages.
   Every language module takes case of providing the correct amount of type informations to the core engine, omitting exceeding features related informations.
2. Every new significant type (types that extend Object?) defined in every language is registered via the unique entry point defined by the core engine.
   The type registration mode should require the maximum amount of information, taking C# as max target reference: a namespace and a class name. (And base class name if it can't be automatically obtained)
   Methods, variables, attributes registration mode should only require the basic amount of information. (Example: take GDScript as target feature set)
   GDScript should support namespaces.
3. Every language module generates their data to manage the new types.
   Examples:
   1. GDScript module makes those types available directly by identifiers in GDScript.
   2. Mono module generates the new types corrisponding assemblies to be used in C#.
   Language "A" that wants to use a type defined in language "B" will delegate the call to the engine, which lets the runtime of language "B" execute the real code.

If this global registration of custom types is not possible or too complex it would be better to consider to not support at all other languages besides GDScript for plugins. GDScript types should however be visible by other languages.
#7402 #17387 #17348 #13357 #6767 #6067

[Cygon]

While I see that you put a lot of thought into this concept and do not want to devalue your input, I believe this design has been abandoned by modern game engines for good reason.

1. Godot scenes already allow creation of custom nodes that can be arbitrarily simple or complex.

The scene tree panel has two buttons:

The first one creates a built-in node, the second one places another scene (= custom node). The window used to select another scene has a favorites list that allows the same two-click workflow possible with built-in nodes.

2. When using inheritance in this manner, you invariably run into the "is a rectangle a special case of a square or is a square a special case of a rectangle" problem.

For example, which hierarchy is more correct?

- Entity
  - FlyingEntity
    - EnemyAirplane
    - PlayerAirplane
  - DrivingEntity
    - EnemyTank

or

- Enemy
  - EnemyAirplane
  - EnemyTank
- Player
  - PlayerAirplane

In one case, you duplicate the Enemy implementation (or write boilerplate code to forward to an external implementation), in the other case you duplicate the Flying implementation.

What modern game engines used to resolve this ambiguity is the "Composition over Inheritance" principle (https://en.wikipedia.org/wiki/Composition_over_inheritance) or more specifially, a "Component/Entity System" (https://en.wikipedia.org/wiki/Entity%E2%80%93component%E2%80%93system).

This allows an entity (here = enemy/player on airplane/tank) to be composed of individual components that make up the entity. Godot has "optimized" its C/ES implementation by making both components and entities the same: nodes. So your entity would be the root Spatial node and the components would be some of the nodes below.

Unreal Engine 4 has attempted to support both C/ES and inheritance and, this is my personal opinion, ended up in a pretty poor place.

[samdze]

Sorry, I think you didn't understand what exactly was described, perhaps the explanation goes too far.

There is no crucial change in the use of inheritance and composition: Godot already uses inheritance to build its hierarchy of nodes and composition would obviously be maintained to build node trees.

The main thing here is: Godot lets the user attach scripts (composition) to mimic the behaviour of inheritance and leaves the responsibility to make it work properly to the user himself.
Solution: hide this complexity to the user and let the engine ensure the proper functioning of the node.

And here we come to the conclusion: from the user eyes he must only be able to define new types of nodes, and NOT new types of scripts to attach to nodes.

Please try to reread the issue, unfortunately it is long, I know, and let me know if you still have any doubts.

[reduz]

Sorry, this was discussed at lenght. There is no reason to hide this complexity from the user, it is in fact intended and good that it's exposed. Nothing is gained from this, it is a node with a script attached, not a new engine datatype. No reason to make it look like something that is not. On Mar 11, 2018 21:26, "Samuele Zolfanelli" <[email protected]> wrote: Sorry, I think you didn't understand what exactly was described, perhaps the explanation goes too far. There is no crucial change in the use of inheritance and composition: Godot already uses inheritance to build its hierarchy of nodes and composition would obviously be maintained to build node trees. The main thing here is: Godot lets the user *attach scripts (composition) to mimic the behaviour of inheritance* and leaves the *responsibility to make it work properly to the user himself.* Solution: *hide this complexity to the user* and let the engine ensure the proper functioning of the node. And here we come to the conclusion: from the user eyes he must only be able to *define new types of nodes*, and *NOT new types of scripts to attach to nodes.* Please try to reread the issue, unfortunately it is long, I know, and let me know if you still have any doubts. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#17418 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF-Z22HT0uiXVyRBoCkPPK4fepG5soLXks5tdcC_gaJpZM4SlXbw> .

[reduz]

This should go to the FAQ at this point...

…

On Mar 11, 2018 21:35, "Juan Linietsky" ***@***.***> wrote: Sorry, this was discussed at lenght. There is no reason to hide this complexity from the user, it is in fact intended and good that it's exposed. Nothing is gained from this, it is a node with a script attached, not a new engine datatype. No reason to make it look like something that is not. On Mar 11, 2018 21:26, "Samuele Zolfanelli" ***@***.***> wrote: Sorry, I think you didn't understand what exactly was described, perhaps the explanation goes too far. There is no crucial change in the use of inheritance and composition: Godot already uses inheritance to build its hierarchy of nodes and composition would obviously be maintained to build node trees. The main thing here is: Godot lets the user *attach scripts (composition) to mimic the behaviour of inheritance* and leaves the *responsibility to make it work properly to the user himself.* Solution: *hide this complexity to the user* and let the engine ensure the proper functioning of the node. And here we come to the conclusion: from the user eyes he must only be able to *define new types of nodes*, and *NOT new types of scripts to attach to nodes.* Please try to reread the issue, unfortunately it is long, I know, and let me know if you still have any doubts. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#17418 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF-Z22HT0uiXVyRBoCkPPK4fepG5soLXks5tdcC_gaJpZM4SlXbw> .

[samdze]

@reduz Could you please explain why it's good to expose the script Object variable to the editor?
There is something useful the user is capable to do with script exposed that is not without?
From what I observed using the engine, what this approach offers more than the hidden one is ONLY the possibility for the user to make mistakes.

And that is because the problem behind everything is that the only purpose of a script is to simulate inheritance.

On the other side, eliminating the concept of "script to attach to a node" to the user eyes, the entire user workflow would be simplified and totally coherent.

[reduz]

Does not seem to be my experience so far or that of most people. Even if mistakes were made, it's by far not a common scenario.

As such, It does not justify changing this.

[samdze]

I think your argument is just not valid.

While it is true that the above situation it's not a common scenario:

1. that's because throughout the entire workflow scripts are often, but not always (incoherency) masked as nodes, and since attaching a script to an instanced node is a rare thing, scripts to be attached to nodes, following your reasoning, are not justified as an additional concept to be learned by the user.
2. you're completely ignoring all the benefits that this simplification would bring to the workflow.
   A single, simple, coherent approach for every possible situation is what a game engine should aim for.

And I completely disagree with:

There is no reason to hide this complexity from the user.
No reason to make it look like something that is not.

The whole point of a game engine is to hide complexity, to abstract a complex system to be able to simulate a world that is convenient and easy to use for the user to make games and applications.

The current system is just putting a spoke in the wheels in the way to such a "perfect" game engine.

I didn't read or found any reason why the current system would be better.
It seems you're not considering this solution only by side taken.
Therefore, I firmly believe that this issue should not remain closed.

[reduz]

I am sorry Samuel, I don't know much about theory of programming, what things are supposed to be fore, and I am really bad at coming up with theoretical constructs or architecture. Therefore, I don't feel qualified to continue this argument.

I only write code based on real-life use cases and what people actually doing games need when doing games. So far I think people likes the stuff I wrote, so I don't feel motivated to change things just for the sake of an hyopotesis I can't quite picture it's advantages as useful in my head.

[reduz]

In any case, if you have something in mind, and want to do a fork to prove of your idea is good, and users like it considerably.over the current system, while using it on their projects, I would be willing to reopen this.

Otherwise there is zero interest in doing any change in this area.

[ghost]

http://docs.godotengine.org/en/latest/about/faq.html#i-have-a-great-idea-that-will-make-godot-better-what-do-you-think

Godot developers are always willing to talk to you and listen to your feedback very openly, to an extent rarely seen in open source projects, but they will care mostly about real issues you have while using Godot, not ideas solely based on personal belief.

Please reformulate the issue as written in the link. Long paragraphs are meaningless without saying what you were trying to achieve and failed.

I have no problem with the current implementation. From what I understand, most things you described can be done cleanly using Scenes, Nodes and Scripts. It is just that the editor can't display custom nodes. I find creating and scripting new type without access to individual node scripts more complex. Also, do I have to create new type every time I want to add a new behavior to a node?

[willnationsdev]

@samdze Based on discussions I've had reduz and others, here's what I think on this.

A while ago, I was working on integrating scripted types into the ClassDB and wanting to implement auto-registration of scripts as they are created. Furthermore, the Object class would enforce runtime script constraints to simulate in-engine-like non-removable script functionality. There would still be an "add script" and "clear script" ability, but clearing would go back to the custom script and adding a script would become auto-derived from the custom script.

What you are suggesting, to completely get rid of the concept of a script on the front-end, actually feels like a step backward. Having a two-layered approach to defining object functionality (the "object" and the "script") still seems fundamentally easier to understand to me. The "object" (Object + custom script) being "what can this do" and the "script" being the "what I actually want to do with it". If I suddenly want to change what I want to do with a given object, I don't wanna have to replace the object entirely, just the directions I am giving to it, i.e. the script that is on the object.

Now, it's been discussed at length that the best solution individual scripting languages should be responsible for handling whatever identifiers they want to identify for themselves. reduz doesn't want to pollute the core with any information that is uniquely useful to a subset of languages (if it goes in script_language.h/.cpp, then it should be something that all languages really need to have defined).

The goals that you and I discussed before, about having custom types not be changeable after creation at runtime, would involve several changes to core in order for them to be viable. This is not something that reduz wants to implement, primarily because the main use case of THIS method is for securing scripted inheritance hierarchies for custom types at RUNTIME which hardly ever needs to be addressed anyway (bare with me on this). The only time when it would be pertinent is if you had a script that, at runtime, attempted to change the script of another node. This would hardly ever happen, with the "exception" being a tool script which happens at design-time and not runtime anyway.

As a result, reduz's suggestion that this be setup as design-time checking in the editor actually makes a lot of sense. It keeps the code out of core and ensures that the editor can prevent us from modifying custom type scripts from their inheritance hierarchies (which IS the goal we had). It also happens to be the least complicated of all of the options available, even if it might seem "hacky" at first glance. But if the editor checking is all that's really necessary (and in large part, it is), then over-solving the problem through additional complexity in the core would actually be a detriment to the engine's cleanliness.

As for creating identifiers available to each language, the opted method is to have the individual scripting language modules handle that on their own using the editor data. So, for example, I plan on adding constants to GDScript's global_map as .gd files are created. This will allow users to define a bullet.gd and then have Bullet type be accessible to them (even from the same script ideally). If we create identifiers of this sort for ANY script (not just scripts of the same type), then you'll be able to access via some global name scripts that have been defined in any language. I intend to do this, although I'll be namespacing them in a rudimentary way (bunch them up into constants on a GDScript that is added to the globals. The key to access the GDScript instance then becomes a "namespace" that groups access to the script's constants which point to other scripts).

The idea would be that identifiers of this sort would be generated on the fly for all .gd files (and possibly .vs files too, to be used by VisualScript). The C# folks are already working on removing explicit file path dependencies, iirc (something about that was said). However, while identifiers may be made on the fly at design time (not run time - not really needed and a lot more complicated), the custom type definitions would need to be manually declared in an EditorPlugin. This actually makes MORE sense since custom types have additional information they require than a simple .gd file can give them (description, icon, documentation, abstract-ness, etc.).

I'm working on all of this in my own fork, branch custom-types-editor.

I think that once I actually create a PR of this sort, we'll start getting a lot fewer questions about it, so if it does go into the FAQ, it may not need to be there long. Most of the reason people ask about it / offer suggestions like this is because the current system's usability is lacking. If you eliminate the usability issues, then the drastic solutions like this (the I once also pursued) won't become as common.

[samdze]

I'll try to refurmulate in a less theoretical way in the next days, but making clear that the theoretical part is important because it highlights the design flaw that Godot has, making the user unnecessarily intervene to ensure the correct use of inheritance of nodes/objects, exposing a totally unnecessary engine component, the script.

First, regarding your doubts, @Noshyaar, most things can be done using Scenes, Nodes and Scripts, that's true, but it can only be done in an inconsistent and fragmented way.

Scenes are stored and accessed through the filesystem, that's ok, they are data, like a large number of other Godot objects.
Nodes are accessed through create window or Node type identifier because nodes are types, classes, not Resource instances (data).

These are the only two concepts Godot needs: Data and Classes.
Custom nodes (and Resources) would be the anchor for the user to create his logic following seamlessly the Godot Object hierarchy that is already there.

Right now there's a third concept, the script, that acts like a mixture of both concepts and does nothing more than mimic inheritance using composition.
So, why use composition to mimic inheritance? That's just a useless indirection. Let's just use inheritance and make the whole system cohesive and simpler.

Custom nodes and resources that request (through import dock: taking advantage of enstablished engine features) to be shown in the proper create window can be shown seamlessly, as they ARE nodes or resources, not a hacky, ad-hoc representation of a proper node/resource.
This makes the entire engine behaviour totally predictable if the user has learned the only concept of the Object. (Node, Resource, etc.)

As of creating a custom type every time you have to add a new behaviour:
you already do this creating a new type of script inheriting another script or a node, there is no conceptual difference.
Only difference is that there is inconsistency if you want to inherit a script or a node in the current system, and there is no hierarchy tracking due to the nature of a script: it's not a new node, (so it's normal that is not shown in the create window) but wants to act like it was: design flaw.

@willnationsdev, my solution doesn't require runtime script constraints. The script variable is simply not exposed to scripting languages, but on C++ side it's still there and accessible.
There is NOT additional complexity in the core.
It's the editor at design time that takes care of managing the internal Object script variable.

So far this would be similar to your solution, BUT, with this new approach there's no need for ANY constraints, even in the editor, because by design there is no script attached that has to be handled, to the user eyes.
What all of this means is: in the editor "Change Type" takes the role of "Attach Script", "Clear Script" and of "Change Type" itself, because the new design brings togheter the old two heterogeneous concepts.
At runtime, a call "change_type" on a node replaces simply every type of operation that was previously managed by several separate operations.
And note: internally, Godot doesn't have to replace the whole node, it can replace just the script if the intention is compatible.

I don't see how one would believe that the current approach of two separate concepts that have to live together to create custom behaviours is simpler than the one I'm proposing.
Attaching a script to a node to extend its capabilities it's a simple concept once the user is introduced to both the two raw materials, it's true. A person already familiar with that has no problems.
But, it's even simpler if the raw material is only one: the Object hierarchy.
Following the class hierarchy that Godot defines by default is the simplest and most immediate way of acting and thinking.

The feeling that should shine through with this new approach is that of each piece that fits perfectly with the others.

Hope I made it clear how my solution differs from yours.

I think that being able to access types defined by any language is a key feature.
For example: there must not be the eventuality that some plugins remain unusable within a project that uses a different programming language.
I believe that each language should fulfill a basic set of features to support this, including namespaces.
Remember: custom types editor-related data would be defined in the import dock, keeping this informations separated from the custom type definition itself.
That's all I proposed regarding cross-language support, much of what you have explained seems to be compatibile with my idea.
However, I think that a better, homogeneous support for less languages is far better than a worse, heterogeneous support for more languages.

[willnationsdev]

@samdze I think I gotcha now. You want to eliminate the concept of scripts in the first place as far as the interface for the user is concerned. And when they create new scripts, they will feel like they are just creating new types in the editor.

So the argument comes down to whether it's good/bad for the concept of scripts to exist at all in the first place, and then whether it'll even be worth the effort to implement all of the necessary changes.

Based on reduz's previous statements, he has zero interest in hiding the existence of scripts from the user. With the editor changes I'll be making, I believe the demand for further changes will be greatly reduced, and the priority will shift to other more immediately desired features. For this to be made, you would need to take the initiative and create a forked version that works the way you describe. There would also be no guarantee that all of that work would actually be used though (I already wasted 2 months of development, so I know the feeling).

No one's shooting down the idea per se. It's just a matter of this being a community project. Ideas are integrated when a majority of users actually want them integrated. Ideas are only implemented when someone takes the initiative to actually make those changes. So, if you can get people interested in the idea (perhaps taking a poll? Asking if people want to do away with the concept of scripts in the editor?), and then you can actually build it and make a demo build of the engine in that way, then we can see if people really prefer the way that works over the master branch version.

Edit: I would get clarification from other core devs first about what kind of implementation approach for this would even be accepted if it DID prove to be desired by the general public though. And be patient with them because they've been receiving a LOT of "pinging" / badgering / messaging about this entire situation for a while, especially the last week or two (I know because I was the one doing a lot of that messaging, lol).

[samdze]

I understand what you're saying regarding the fact that this is a community project.
However, we are now talking about a matter that has the simple creation of custom nodes only as a side effect. I think your implementation only builds on top of the current, non-optimal system (reasons explained below, again) so it may seem the easiest and most immediate solution, but it is something that will surely make things worse in the long run.

The problem of this situation is that this is for the most part a improvement that is highlighted in the theoretical field. Normal, inexperienced users are not able to imagine an overview of the entire workflow only presenting the theory, the proof is that they often propose solutions that would unbalance the whole engine, which would only add more exceptions or special cases, or that are simply the result of a naive desire of a particular feature.
This is not an excuse. It's just to make it clear that sometimes developers have to take matters into their own hands knowing what's best for users.

There isn't a particular aspect of the workflow that is greatly improved, it's more about a very wide range of workflow parts that are improved, the more or the less depends on the part itself.

I'll try to explain why this new approach is certainly better than the previous one from a theoretical point of view and why I think it's totally worth the effort. Hope my next explanation will make you think.

Engine complexity

Right now we have two distinct sets of entities, objects and scripts to attack to objects.
Suppose now that entities objects have X modes of use and entities scripts to attack to objects have Y modes of use.
Let's assume that each mode of use taken individually is valid for each entity.
Combinations of the uses of the two entity sets let the user define its own logic.
Those combinations are logically an amount of X * Y, let's keep these in a set called Q of X * Y elements.
In the middle of all of this there's the first problem: in practice, some of these combinations of use are not valid. An example is a script that inherits from an object that is not in the hierarchy of the object it is attached to.
The existence of these invalid combinations is given by the nature of a script itself: the fact that it makes use of composition to simulate inheritance leaves some unwanted side effects: it adds use modes derived from the fact that it's not pure inheritance, which are only undesirable ones.
Let's name this set of invalid combinations Z, subset of Q.
Now, not only the engine/editor has to handle each mode of use of both entity types, but it also has to handle the X * Y possible combinations. This increases complexity, as it increases the number of exceptions, of special cases, of checks to be made, of restrictions the engine has to impose.
All those restrictions and checks are there to ensure the Z invalid combinations are avoided or notified to the user when they appear.
This sort of extra handling is only expected to increase as new engine features are implemented and integrated in this system, or at least those new features will have to take care of it. Theoretically, this is extra work to do for each addition that is built on top of the current system.

Let's consider the new approach:
the system makes the user able to create new types of entities belonging to the objects set thanks to inheritance. Woah, there is no need of the extra scripts to attach to objects set, so the total amount of modes of use is only X+A = W.
X being the amount of uses of a not directly inheritable object entity, and A being the extra uses unlocked by the ability for the user to inherit directly from object entities.
The engine can make safe assumptions and now, the amount of invalid combinations is 0, because the combinations themselves are 0, everything is generated by a single entity set.
The new approach uses "proper" inheritance, every use case is valid, because they all follow the intended behaviour.
But, what about the A extra modes of use of the objects set?
Fact is: A is a subset of the already defined set Q.
A is a set that keeps only the desirable use cases from Q, because, as already stated, A is the result of pure inheritance, eliminating at the root every kind of possible bad use case Z.
That means there's less complexity, the engine execution can flow like a charm without ever have to be worried about what the user may do wrong.

I think it's crystal clear that the new approach would keep the code of the engine core and editor clean and ensure a simpler implementation of high-level features in the future.

User workflow

Let's now talk about user workflow.
The same argument applies here too, less "atomic" operations, possibility of confusion or of making mistakes totally rooted out.
But let's make a more user-focused demonstration too.

As already stated there will be no need for a second set of entities, scripts to attach to objects, when now the total amount of things the user can do is Q, then they would be W, with a major difference:
no combinations are involved.
Combinations of concepts make the whole process less straighforward, less intuitive and in need of more effort from the user, that has also to learn both of them first.
Although combinations of distinct types of entities in general can be very powerful, they are only if those would expand the possibilities given by the engine at a relatively low cost of complexity.
It's clear that this isn't the case.
The user is already driven to learn how the Object hierarchy works to be able to create any kind of custom behaviour. The new system does not need any other knowledge, unlike the current one.
Keeping a single type of entity the user has to cooperate with, the engine editor can keep its UI simple and straightforward, every possible operation is performed on objects, no ambiguity, maximum consistency, the user perfectly foresees how things will work in every case.

The editor of the old system could get near the same immediacy only adding a range of shortcuts that let the user skip redundant or superflous parts of the workflow, one for each mode contained in set A, thus increasing complexity, confusion and the amount of options that nearly do the same thing and still, the system would be susceptible to incorrect user modifications of the setup prepared by shortcuts.
Would you like to prevent it by adding other restrictions and special cases? Complexity.
The user will not know anymore when and where he can do his business freely.

Conclusion

It's not easy to envision the advantages at first glance, intuition is often necessary and that's why people frequently say "I don't see how this would be better or different than how it is now", but this theoretical demonstration should clarify once and for all that there IS a difference, maybe not too big in that particular workflow section you're thinking of, but it's HUGE if the whole workflow is taken into consideration, both for the user and the developer.

Hope this explanation is useful for you @willnationsdev, @reduz and the other core devs to better understand what I'm trying to achieve and convey.

[Type1J]

I think that this issue is purely aesthetic.

Would it be satisfactory to define new node types based on scenes listed as such in the project file (The "+" button would list them as a "built-in type")?

[willnationsdev]

The problem of this situation is that this is for the most part a improvement that is highlighted in the theoretical field.

As mentioned, the devs only really plan to devote time to features that will provide a practical change in functionality or user experience. There's a reason it's in the FAQ.

Normal, inexperienced users are not able to imagine an overview of the entire workflow only presenting the theory, the proof is that they often propose solutions that would unbalance the whole engine, which would only add more exceptions or special cases, or that are simply the result of a naive desire of a particular feature. This is not an excuse. It's just to make it clear that sometimes developers have to take matters into their own hands knowing what's best for users.

That's why we have GitHub Issues, to allow devs who do know what they are doing to filter these suggestions and ensure that "exceptions, special cases, [and]...naive desires" become integrated into the engine on a minimal basis, preferably not ever.

Those combinations are logically an amount of X * Y, let's keep these in a set called Q of X * Y elements.
In the middle of all of this there's the first problem: in practice, some of these combinations of use are not valid. An example is a script that inherits from an object that is not in the hierarchy of the object it is attached to.
The existence of these invalid combinations is given by the nature of a script itself: the fact that it makes use of composition to simulate inheritance leaves some unwanted side effects: it adds use modes derived from the fact that it's not pure inheritance, which are only undesirable ones.
Let's name this set of invalid combinations Z, subset of Q.
Now, not only the engine/editor has to handle each mode of use of both entity types, but it also has to handle the X * Y possible combinations. This increases complexity, as it increases the number of exceptions, of special cases, of checks to be made, of restrictions the engine has to impose.

Well, yes, in theory, it would be awesome if the Y types could be automatically relied upon to always have a proper relationship to the type X (and thereby virtually eliminate the need to conceptually separate the two at all), but there is a critical problem with your analysis here:

Scripts that users define are, by definition, user input. It will ALWAYS be possible for a user to supply a bad input to a program, some input that is not compatible with the program's expected input format or a valid input that does not connect properly with the program's data. This is a basic aspect of computer science. The reason objects exist separately from scripts in the ClassDB is precisely because objects are developer-defined and unchangeable while scripts are user-defined and changeable. And if it is changeable, then the program needs to examine the changes to confirm whether they are compatible with the developer-defined information. There is literally NO WAY to enforce that a given "user-written object-definition" will conform to the standards presented by the engine unless validations on that text file are performed.

If the user says they want to define a new object that derives from x of set X, then we inevitably have to confirm whether x is IN X and whether the things they are attempting to use in their user-defined object is something that x can do.

Even if you did take away the user interaction of attaching/removing scripts and made it seem to the user as if they were simply defining new classes in the engine, you would still have to deal with an equivalent level of complexity at design-time with the editor checking whether the user-defined input appropriately matches the developer-defined data. So, while you think this would simplify things a lot, even in theory, you are still not actually changing the problem that you are trying to solve, and it isn't really a problem that CAN be solved simply by virtue of the fact that Godot Engine decides to be a friendly engine and lets people define their own object types already, via the power of scripts.

Combinations of concepts make the whole process less straighforward, less intuitive and in need of more effort from the user, that has also to learn both of them first.

Literally every game engine supports this though. Users have to learn how to use the engine's API and then learn how to create scripts, attach them to engine code, and then use them together. UE4, Unity, GMS, they all do this. And I don't think "[learning] both of them" is really a significant obstacle for any actual users out there.

the engine editor can keep its UI simple and straightforward, every possible operation is performed on objects, no ambiguity, maximum consistency, the user perfectly foresees how things will work in every case.

There's no evidence that ANYONE is getting confused about "how things will work" when they attach a script to an object. At least, I haven't seen anyone asking things like, "What does it mean when I attach a script extending BaseButton onto a ToolButton? What happens if I move it to a TextureButton?", "What happens if I remove a script?", or "What happens if I add a script?". They are all very basic concepts that all beginner programmers become comfortable with as they learn about inheritance and how scripting (in any game engine) works.

Edit: "no evidence that anyone is getting confused about [this]", specifically in regards to how Godot works.* When questions like this DO show up, they are typically basic, fundamental questions about how inheritance in programming works in general. So they have nothing to do with any need to change Godot's scripting system.

If we did remove the concept of scripts from the UI, but users could still somehow define types via scripts, then what you would ACTUALLY get is more confusion since users might click on a node and wonder where its functionality is coming from or where it's defined. Or if it automatically redirected them to the relevant script or to documentation depending on whether its was a scripted type or a C++ object, then they'd end up getting annoyed since they'd want to be able to tell, at a glance, whether they would be sent to one or the other. And then we'd just go back to having the script icon, again, so that they could click on it and be taken directly to the scripted definition of the class that the node is a member of.

So no, I don't think removing the script icon entirely is a good idea (and this has also been discussed at length in various places).

It's not easy to envision the advantages at first glance, intuition is often necessary and that's why people frequently say "I don't see how this would be better or different than how it is now", but this theoretical demonstration should clarify once and for all that there IS a difference...

Actually, it's very easy to envision how the object vs. script data type changes you describe would be advantageous if they were implemented (we would be able to dissect user input and automatically have users' scripted ideas just be processed validly, successfully, auto-magically). Unfortunately, they are not physically possible for all the reasons I mentioned. It WOULD be different from how it is now, but the parts that CAN be done (UI updates) would actually make things more confusing because of the incongruity it would make with the things that CAN'T be done (wouldn't make sense to create UI changes that don't reflect engine changes properly, obviously).

What IS possible to do though is have the engine or the editor run checks that a script is validly inheriting from and using an object or script. It was deemed that performing runtime checks in the engine core would be code bloat and ultimately unnecessary, but that performing design-time checks in the editor would be a good idea. So that is the solution we are implementing.

Now, if you can somehow outline a concrete set of implementation steps that would leave the theoretical field and give us practical, executable actions that would fulfill the awesome ideas you are having, I'm all ears. Would be a pretty groundbreaking achievement in the field of computer science, if I'm understanding correctly (apologies if I somehow missed the boat on what you're meaning all was).

[willnationsdev]

@Type1J I actually was thinking about setting this up, but I didn't think people would necessarily want it since you'd effectively be putting the chain link functionality inside of the + functionality when technically speaking they do completely different things. I'll have to create a separate issue about it and gauge people's interest.

[willnationsdev]

@Type1J Oh, apparently it already exists as two separate Issues talking about the same thing. 10855 and 3895.

[samdze]

@willnationsdev I think you misunderstood me.

I thought it was clear that my explanation only considers the characteristics of the high-level context we are dealing with.
Lower-level checks, or checks made by other components of the engine, like script parsing and validation are obviously there to stay, because as you said, scripts are unpredictable user input.
This reflects the fact that the engine core doesn't need any major change (maybe only the "ScriptDB" type registration interface and the hiding of script to scripting languages?)

Ascertained at this point that the modifications would nearly only affect the editor with the exception of the custom type registration feature that is already in development, what would the implementation impediments be?

Anyway, Godot already combines different concepts like other engines do. Node/Object hierarchy and composition in tree structures, and I really like how it does it.

Even Unity, for example, gives the user a unique entry point to interface with the engine, the ability to define custom Components. That's different from the current Godot approach, which would let the user define scripts to attach to existing components in Unity's context. And I don't know, but maybe internally they are handled exactly like Godot goes with Object+Script.

Oh, regarding the script icon, I refer you to the original issue post:

5. The editor UI can be left almost as it is. The script icon to the right of a node indicates that is an instance of a Custom Node Script, and clicking it will bring to the script editor of that Custon Node Script.
   Only difference is: that "script" cannot be removed, because to the user eyes that is the type definition of the node, much like you cannot remove the KinematicBody-ness from a KinematicBody.
   If the user wants to remove/change the behaviour of a node, he'll have to change its type. The UI could support the quick creation of a Custom Node Script extending the selected one. (Like how it works now with Godot Scripts)

No more "Attach Script" and "Clear Script" on right click on a node in the SceneTree dock (nor next to the "filter nodes" bar, maybe to be changed with a Change Type button). The "Change Type" option will be the only necessary one.

"Script" will only be a name to refer to the user's class definition files.

[willnationsdev]

Wait a second. What you are saying about the script icon at the end there (I just realized) is exactly what I'm already planning on doing with the design-time checking of custom types in the editor. So, as far as custom types are concerned, they WOULD be considered new types of nodes essentially. That was the whole reason for the debates in like 3 other Issues you referenced. The scripts come auto-attached, but they will give a transparent icon to indicate that the script is a "part" of the definition of the node, i.e. the script cannot be removed at design-time. And if you attempt to extend it, it will automatically start extending from the custom type script.

And you are slightly wrong about the comparison to Unity, in my opinion anyway. Godot has you define new extensions for nodes, yes, and you then must attach those script extensions to the given node, but Unity does this as well. You define a custom Component, and then must attach that component to a GameObject as a script. It has a slightly different flavor, but because of the way scripts are organized (1 script per node, directly attached as opposed to a MultiScript system), there isn't any other way of doing it.

I think all in all, you have been overly concerned with a theoretical set of changes without realizing that the changes we are currently implementing are the practical implementation of your theory in action already.

I am also attempting to fix the "editor icon remains after removing plugin" bug and add the "allow scenes as custom types" feature. So it'll be a super PR that knocks out about a dozen Issues all at once pretty much.

[samdze]

My set of changes provides for the complete elimination of the script to attach to an object approach to make the workflow uniform and more manageable.

At this point, why keep the old approach that does not define "directly inherited" custom types?
The system you're implementing, if I understood correctly, would be able to do everything the old "script to attach to a node" system can do and more, the more being "actual" custom objects/nodes/resources.

Why keep the bloat, the complexity, and the incosistency that the old system together with the new one would bring?
I think that's worth the breaking change.

Unity uses the classic ECS approach, the GameObject is by design only a container immutable in itself if not in what it contains, not even the Unity engine core extend or customize its behaviour. Since Godot merges the concepts of entity and component in a single one, the node, and composes them in tree structures, they are expected to be the directly extensible components of the engine.

[reduz]

@samdze As I mentioned before, I understand your point perfectly, but still don't share the vision. You will simply not convince me with words.

The only way you can convince me is to make it happen by yourself (I wouldn't expect anyone to work in your place to make this happen) and show us that people actually prefers your approach over the current one... which so far no one has really complained about (even though you insist that it's broken).

[samdze]

Yes, understood, that's clear.
Now, since @willnationsdev was implementing a feature that would be necessarily needed in the full version of the new system, I would like to see if it is possible and ok for him to come together and join the efforts.

[willnationsdev]

@samdze

Why keep the bloat, the complexity, and the inconsistency that the old system together with the new one would bring?

In addition to reduz's statement, consider this.

Creating a custom type relies on there being a dedicated file for the type (so built-in scripts don't work). If we mandated that ALL scripts created had to be custom type scripts, then we would also be forced to take away the convenience that built-in scripts provide.

In addition, creating custom types involves supplying an icon to the editor. Are we going to tell people that in order for them to make a new type, they must also create an icon to go with it? What if they don't want to make a new icon? If we just re-use the icon from the inherited class, then we end up with multiple, "completely different classes" that have the same identifying icon. This would be bad UX in general.

Another thing: custom types have the option of having XML documentation defined for them. If we attempted to automatically generate XML documentation for every script a user created to simulate them as in-engine types, then THAT would be code bloat since people will often create scripts at the drop of a hat.

Point being that not all scripts need to be as "high priority" of an item as a new class with all the fancy bells and whistles that go with it.

And I also don't share that specific design philosophy. I do think that scripts should remain separated from objects ultimately, so I wouldn't be interested in "[coming] together [to] join the efforts" you would like to invest in your vision.

I do wish you luck in demonstrating your own version of it though.

[samdze]

I see what you mean and I understand your concerns.

However, I would like to clarify some things.

This system doesn't assume that every custom type is displayed completely as an engine type.
By default, a newly created custom node/resource is not shown in the corresponding "Create New" window.

3. Godot generates a default .import file for that Custom Node Script, extrapolates superclass info, and defines a default icon (default icons for each letter of the alphabet or a set of utilizable pre-made icons would be great), description, whether the node should be displayed in the "Create New Node" window (maybe off by default) etc.
   Data only relevant to the editor and little more.

Custom types can be defined as built-in into a scene, and in case of a simple implementation, yes, the icon should stay the same of the inherited type, the node can't be displayed in the "Create New Node" window and the user would not be able to customize the editor data, including documentation, but that's fair, as they are types that are supposed to support that scene only.

Things change if the built-in type is the root node of a scene:
this is where support for custom nodes that are in fact scenes comes in.
The scene file is treated like a script by the import dock, and the user can define the usual editor data.

Support for the creation of custom engine-like nodes that are in fact scenes is natural and intuitive.
Scenes created with a built-in root Custom Node Script would automatically be processed like Custom Node Script files by the import dock.
Having the Custom Node Script built-in into the scene makes sure the Script (Node Type) is owned only by that specific scene, making Node Type -> Scene a unique correspondence.
Such scenes become accessible and instantiable through filepath, through "Create New Node" window and through (custom node) identifier.
This would be a very powerful feature, perfectly integrated into the new system.

[Shadowblitz16]

@akien-mga Can this be reopened?
Both are good but the issue is only 1 script can be attached.
We should be able to create new types through inheritance while attaching multiple scripts to nodes.

[akien-mga]

This should be discussed on godot-proposals, not here.