Add occlusion culling based on proxy/tester meshes

[SlugFiller]

Describe the project you are working on

3D game

Describe the problem or limitation you are having in your project

While Godot already has a few occlusion culling solutions, they are all designed for specific scene shapes. I offer a simple occlusion culling method that is more general, allow for a greater range of scene compositions, and can be performed primarily in GPU. Furthermore, it works by an opt-in rather than opt-out method. i.e. Objects are hidden by default, and are "revealed" by the occlusions computation, instead of being visible by default, and being "hidden".

For this method to enjoy GPU acceleration, the renderer must support two features:

1. Render to buffer A while clipping using depth buffer B.
2. Object index rendering. i.e. rendering using integer colors, with no anti-aliasing, so that the exact same values can be accurately read back from the buffer.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The idea is simple: Have specific "test" meshes that are rendered to a second buffer. If the test mesh is visible, it adds certain target objects to the render. The test mesh can therefore be any simple mesh that fully contains the target objects. It can be a room, it can be a simple bounding box around a more complex mesh, or it can even be a box covering half the world in an oct-tree setup.

The tester mesh and its contained targets have a many-to-many relationship. A room can contain multiple objects. But an object can also be in a transition between two rooms. The visibility works using or - if an object is "inside" multiple rooms, and only one of the rooms is rendered, the object is rendered.

Tester meshes can, themselves, be inside a target object, creating a "testing hierarchy". An example usage of that is an oct-tree. The root tester nodes are axis-aligned cubes that split space into 8 equal regions. The target of each tester is a parent to 8 more tester cubes, farther splitting that area. And so on, and so forth.

An important note is that the choice of which object is handled by which tester, i.e. which object is in which room, is NOT automatic. It is handled programmatically by the game's script. The appropriate hierarchy can therefore be carefully designed per-game.

As an added bonus, this can double as frustum culling, as a tester object that is behind the camera would not be rendered.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

We define two classes:

extends Spacial
class_name OcclusionTesterMesh
export var mesh : Mesh
func add_target(target : OcclusionTestTarget) -> void
func remove_target(target : OcclusionTestTarget) -> void
func set_force_active(force : bool, viewport : Viewport = null) -> void
func was_active_last_frame(viewport : Viewport = null) -> bool

extends Spacial
class_name OcclustionTestTarget

set_force_active allows an OcclusionTesterMesh to become active without its mesh being rendered or tested. This can be used when the camera enters a room, since the mesh might not appear if rendered from the inside, so forcing it active on that frame is simpler. The optional viewport parameter allows forcing activation for a specific viewport. Different viewports may render the same scene from different camera angles, so whether the camera is inside a certain tester can vary by viewport. By default, force activation is set for the main viewport.

was_active_last_frame allows having certain activity depend on whether a certain room or object was visible or not. This can be used to turn off AI in hidden areas, or turn certain particle effects or physics calculations on and off. The optional viewport parameter is useful if a viewport renders the same scene from a different camera angle. Which tester objects are active may vary based on the camera.

The rendering process is as follows:

1. For each OcclusionTesterMesh that is forced active, set all matching OcclustionTestTarget as active.
2. Render all opaque objects that are not descended from an inactive OcclustionTestTarget, and are not an OcclusionTesterMesh, into buffer A.
3. Render each inactive OcclusionTesterMesh that is not a descendant from an inactive OcclustionTestTarget using object-index colors into buffer B, while clipping using depth buffer A.
4. Check the colors in buffer B, and activate any matching OcclusionTesterMesh
5. For each activated OcclusionTesterMesh, add all matching OcclustionTestTarget to the active set.
6. For each new active OcclustionTestTarget, render any opaque descendant that is not descended from an inactive OcclustionTestTarget, and is not an OcclusionTesterMesh, into buffer A.
7. Clear buffer B.
8. Render each inactive OcclusionTesterMesh that is not a descendant from an inactive OcclustionTestTarget using object-index colors into buffer B, while clipping using depth buffer A.
9. Repeat steps 4-8 until buffer B is completely black/empty.

Note that the same OcclusionTesterMesh may be rendered multiple times. This is because it might be obstructed by another OcclusionTesterMesh, which may no longer obstruct it after it has become active. The Mesh associated with an OcclusionTesterMesh should therefore be as simple/low-poly as possible, to reduce the cost of rendering it multiple times.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No. This has to be part of the render pipe-line.

Is there a reason why this should be core and not an add-on in the asset library?

Part of the render pipe-line.

[Calinou]

With raster occlusion in master, occluder meshes in 3.x and portals and rooms in 3.x, I don't think we should focus our efforts on adding a fourth occlusion culling solution to the engine.

Instead, we should focus on improving the existing solutions. Did you try any of them on your current project? If so, which limitations did you reach?

[SlugFiller]

@Calinou The issue with these methods is that they're opt-out. They operate by having a simple occluder, and a complex occludee. The suggested method flips things - the ocludee is a low-poly tester mesh, while the occluder can be an arbitrarily complex depth-buffer render.

The existing methods are not general, and do not allow fine-grained control. It's possible to implement a rooms-and-portals system using tester meshes, but it's not possible to implement oct-tree, or bounding-box based testing using room-and-portals. Rooms do not allow hierarchy. You can't have a room in a room. The same with occluder meshes. You can't test a group of objects via a single containing bounding box, then test individual objects within.

You can't, for example, implement a MineCraft clone with the current set of occluders, since you can't exploit the oct-tree hierarchy. But you could do it with the suggested system.

[fire]

Can we check if we have object-index colors buffers? I don't think we have a geometry buffer in Godot 4.

[OctagonalHexy]

This may be A bit off topic but I was watching A video by Bastiaan Olij from 2019 and he mentioned something like this was being proposed for Godot 4 so I ended up here looking to see if this was implemented, but I couldn't find the proposal he mentioned.

He explains it as just A way of grouping occludes manually to prevent burden on frustum culling in dense scenes, but then goes into detail that sounds a lot like this proposal.

at 2 mins: https://www.youtube.com/watch?v=QXUCGzJUfkc&t=120s

I think this is A great idea that should be implemented eventually.