Godot suffers slow massive when there is at least 3k of gridmap

[Kakcalu13]

Godot version

3.5

System information

Windows 10, Linux 20.04, Docker, html5, and mac intel

Issue description

So, I was looking into how to improve the optimize with massive nodes without impact the performance. I tried gridmap and multimesh. They are doing very similar. I'm on GLE2.

See here for example:
bug.zip

The idea is almost exactly same.

See here for example where it worked very well on Mac M1.

Screen.Recording.2022-09-29.at.11.16.53.AM.mov

So when it comes to HTML5, windows, mac intel or linux...it suffers slow and heavy lag (just like the example I sent you)

I can't rely on M1 as not everyone has M1.

So, I looked into this issue: #18673, the issue is almost exactly same as his issue. Apparently, it's fixed. I don't see it on Godot3.5 though, is it in godot3.5 or is it reverted?

Steps to reproduce

1. Load it in godot
2. click play
3. move the camera (ASDW, ARROWS KEYS) to see the massive lag

Minimal reproduction project

bug.zip

[Kakcalu13]

If this is expected, what is the best alternate to gridmap/multimeshinstance node? Like the most lowest cpu cost with no physics, simply 1x1 cube, and plain color

[Calinou]

Try decreasing the GridMap's octant size to 1 to make the mesh rebuilding cost cheaper. That said, GridMaps are not designed to have their geometry changed often.

So, I was looking into how to improve the optimize with massive nodes without impact the performance.

Look into Animating thousands of fish with MultiMeshInstance (GLES3 + GLES2) and and Controlling thousands of fish with Particles (GLES3 only).

If you are looking to create a pseudo-3D extrusion effect, you could also display several Sprite3Ds with the same texture, but a slight offset for each instance. Make sure Alpha Scissor is enabled to avoid transparency sorting issues.

Another alternative is to use raymarching to draw voxels, like the author of #65307 is doing. When doing so, the GPU will be drawing using a fragment shader that writes to depth, without generating any geometry. Therefore, voxel updates are pretty much instant.
However, in 3.x, you won't be able to have working shadow rendering with this (neither cast nor receive). MSAA antialiasing also won't affect raymarching shaders (but FXAA will). See https://github.com/Zylann/godot_sdf_blender for an example of doing this in 3.x.

Apparently, it's fixed. I don't see it on Godot3.5 though, is it in godot3.5 or is it reverted?

This doesn't mean your particular issue was resolved, as it's likely caused by something entirely different.

[lawnjelly]

So, I looked into this issue: #18673, the issue is almost exactly same as his issue. Apparently, it's fixed. I don't see it on Godot3.5 though, is it in godot3.5 or is it reverted?

Note .. that issue is from 4 and a half years ago. The engine has changed quite a bit since then.

As @Calinou says, that may only be indirectly related to your issue. Regarding slow deletion - the most current issue is #61929 which is still open. It is fixed by #62444, but in a PR meeting @reduz said he doesn't totally understand the problem, so that has delayed the fix being integrated.

Also see #65581 which also targets these problems with large numbers of nodes, but again is awaiting review.

[Kakcalu13]

After reading all PRs you mentioned, I am convinced it's due to how we coded, I think. In this issue, it used while rather than for loop and I believe for is faster than while.

Look into Animating thousands of fish with MultiMeshInstance (GLES3 + GLES2) and and Controlling thousands of fish with Particles (GLES3 only).

I tried that too and it lags me. For sure, it's better than gridmap but it lags regardless.

If you are looking to create a pseudo-3D extrusion effect, you could also display several Sprite3Ds with the same texture, but a slight offset for each instance. Make sure Alpha Scissor is enabled to avoid transparency sorting issues.

Another alternative is to use raymarching to draw voxels, like the author of #65307 is doing. When doing so, the GPU will be drawing using a fragment shader that writes to depth, without generating any geometry. Therefore, voxel updates are pretty much instant. However, in 3.x, you won't be able to have working shadow rendering with this (neither cast nor receive). MSAA antialiasing also won't affect raymarching shaders (but FXAA will). See https://github.com/Zylann/godot_sdf_blender for an example of doing this in 3.x.

I'm going to try those!

As @Calinou says, that may only be indirectly related to your issue. Regarding slow deletion - the most current issue is #61929 which is still open. It is fixed by #62444, but in a PR meeting @reduz said he doesn't totally understand the problem, so that has delayed the fix being integrated.

Also see #65581 which also targets these problems with large numbers of nodes, but again is awaiting review.

Gridmap and multimeshinstance doesn't use queue_free() but I'm excited to see them merged as I know I will need them in near-future!

Question for you guys, is there any node that is very very lightweight than gridmap and multimesh? I'm thinking maybe sprite3d would do but at the sametime, I have doubts because I think sprite3d is one of few high cpu cost.

[lawnjelly]

Question for you guys, is there any node that is very very lightweight than gridmap and multimesh? I'm thinking maybe sprite3d would do but at the sametime, I have doubts because I think sprite3d is one of few high cpu cost.

See #61568 . Once that is in you can merge gridmaps to a single MeshInstance. Gridmaps use MultiMesh under the hood if I remember right, but this is far less efficient on GLES2. But I am not very sure of what you are attempting to do in your video, as to whether this will be a good solution for your case.

Generally especially in GLES2 if you need a lot of drawcalls, performance will be poor.

[Kakcalu13]

See #61568 . Once that is in you can merge gridmaps to a single MeshInstance. Gridmaps use MultiMesh under the hood if I remember right, but this is far less efficient on GLES2.

If GLE3 performs better than GLE2, I'm open to switch back to GLE3. I look forward for it to merge one day! Love Godot as it is updating itself non-stop!

But I am not very sure of what you are attempting to do in your video, as to whether this will be a good solution for your case.

I was trying to have a stream data from the camera on robot to godot with neurons (gridmap in this case). I'm re-designing the multimeshinstance again. I think I will also share the zip again once it lags.

My CPU is intel i7 with 258 gb (130 gb space available) on Linux, but I've tested it on other than my laptop like windows 10, mac intel, or others. Only Mac M1 is able to handle massive data.

I'm trying sprite3D and multimeshisntance now. I will bring on a new post in few hours from now on with as much as information as I can to provide using sprite3d or multimeshinstance on over 3k+ nodes

[Kakcalu13]

MultiMeshinstance fixed the issue. It was able to load massive (tested between 3k to 15k cubes using multimeshinstance)

I'm not sure why it was massive lag yesterday...

Glad it's working but it required a lot of tuning. Thankfully, it is not much re-work on the code. See the video,

vid1.mp4

It needs a lot of tune on my end. That's ok.

Although it resolved my goal, this issue is now feeling like a duplicated of #61568 since this issue is more asking how to reduce performance using at least 3k or above gridmap. Do you think we should close this issue as a duplicated?

[Zireael07]

Defeinitely not a dupe of #61568 since that's a PR not an issue

[Kakcalu13]

Closing since there is no action needed for this. This is turning into QA godot now.

P.S. multimeshinstance kept located itself in wrong location over and over unlike gridmap. I wonder if it is possible for it to behave like gridmap exactly using the location. I guess this is another QA godot subject.

I'm so excited for the #61568! Thank you everyone for your time!