[TRACKER] Benchmarks to create

[Calinou]

⚠️⚠️NOTE⚠️⚠️ This list is outdated, please refer to the following one instead:

#36

⚠️⚠️END OF NOTE⚠️⚠️

This is a list of benchmarks to add to the benchmark suite. See the contributing guide for instructions on adding new benchmarks.

I plan to tackle some of those benchmarks personally, so please comment on this issue if you plan on working on new benchmarks (to avoid working on the same benchmark at the same time).

Feel free to suggest new benchmark ideas as well 🙂
Remember that we'd like to keep the project fast to clone, so the repository size needs to remain low. This means we can't do things like adding large detailed 3D scenes with textures, as these usually require several GBs of storage.

2D Rendering

• 50 Sprite2Ds Add Sprite2D benchmarks #18
• 500 Sprite2Ds Add Sprite2D benchmarks #18
• 5,000 Sprite2Ds Add Sprite2D benchmarks #18
• 10 Polygon2Ds (32 points circle) Add Polygon2D benchmarks #19
• 100 Polygon2Ds (32 points circle) Add Polygon2D benchmarks #19
• 1,000 Polygon2Ds (32 points circle) Add Polygon2D benchmarks #19
• DirectionalLight2D without shadows Added PointLight2D and DirectionalLight2D benchmarks #60
• DirectionalLight2D with shadows Added PointLight2D and DirectionalLight2D benchmarks #60
• 50 PointLight2Ds without shadows Added PointLight2D and DirectionalLight2D benchmarks #60
• 50 PointLight2Ds with shadows Added PointLight2D and DirectionalLight2D benchmarks #60
• GPUParticles2D (few nodes with lots of particles) Add GPUParticles2D and CPUParticles2D benchmarks #62
• GPUParticles2D (many nodes with few particles) Add GPUParticles2D and CPUParticles2D benchmarks #62
• CPUParticles2D (few nodes with lots of particles) Add GPUParticles2D and CPUParticles2D benchmarks #62
• CPUParticles2D (many nodes with few particles) Add GPUParticles2D and CPUParticles2D benchmarks #62
• Small/simple TileMap
• Small/simple TileMap with lights
• Large/complex TileMap
• Large/complex TileMap with lights

3D Rendering

Some of those benchmarks should have "exclude on Forward Mobile" and "exclude on Compatibility" properties so that they are skipped on unsupported renderers.

• 50 BoxMeshes
• 500 BoxMeshes
• 5,000 BoxMeshes
• 10 SphereMeshes
• 100 SphereMeshes
• 1,000 SphereMeshes
• 10 skinned character meshes
• 100 skinned character meshes
• 1,000 skinned character meshes
• 10 ReflectionProbes (up close)
• 100 ReflectionProbes (far away)
• 10 Decals (up close)
• 100 Decals (far away)
• DirectionalLight3D without shadows
• DirectionalLight3D with shadows
• 50 OmniLight3Ds without shadows
• 50 OmniLight3Ds with shadows (non-PCSS)
• 50 OmniLight3Ds with shadows (PCSS)
• 50 SpotLight3Ds without shadows
• 50 SpotLight3Ds with shadows (non-PCSS)
• 50 SpotLight3Ds with shadows (PCSS)
• GPUParticles3D (few nodes with lots of particles)
• GPUParticles3D (many nodes with few particles)
• CPUParticles3D (few nodes with lots of particles)
• CPUParticles3D (many nodes with few particles)
• Volumetric fog (global)
• Volumetric fog + 10 FogVolumes
• Volumetric fog + 100 FogVolumes

To compare the relative performance impact of each setting, these tests should render the same scene but with different settings:

Antialiasing

• No AA
• FXAA
• TAA
• 2× MSAA
• 4× MSAA
• 8× MSAA
• TAA + FXAA
• TAA + 2× MSAA
• TAA + 4× MSAA
• TAA + 8× MSAA

Global illumination

• No GI
• SDFGI (default settings)
• SDFGI (8 cascades + lower cell size)
• SDFGI (half resolution)
• SDFGI (half resolution, 8 cascades + lower cell size)
• VoxelGI
• VoxelGI (half resolution, same scene as SDFGI)
• LightmapGI (indirect only, non-directional)
• LightmapGI (indirect only, directional)
• LightmapGI (fully baked, non-directional)
• LightmapGI (fully baked, directional)

Post-processing

• None
• SSR
• SSAO
• SSAO (half resolution)
• SSAO + SSR
• SSIL
• SSIL (half resolution)
• SSIL + SSR
• SSAO + SSIL
• SSAO (half resolution) + SSIL
• SSAO + SSIL (half resolution)
• SSAO (half resolution) + SSIL (half resolution)
• SSAO + SSIL + SSR
• Glow
• Auto exposure
• Fog (non-volumetric)
• Tonemap (linear)
• Tonemap (ACES)
• Adjustments (no texture)
• Adjustments (1D LUT)
• Adjustments (3D LUT)

Core

• Create 100,000 nodes (flat)
• Create 100,000 nodes (nested as deep as possible)
• Create + Rename 100,000 nodes (flat)
• Create + Rename 100,000 nodes (nested as deep as possible)
• Create + Remove 100,000 nodes (flat)
• Create + Remove 100,000 nodes (nested as deep as possible)
• Parse 100,000 Expressions (simple)
• Parse 100,000 Expressions (complex)
• Parse 100,000 Expressions (complex, with function call)

GDScript

https://github.com/godotengine/godot/tree/master/modules/gdscript/tests can be helpful for inspiration.

• Create 1,000,000 untyped variables
• Create 1,000,000 typed variables (explicit type)
• Create 1,000,000 typed variables (with type inference)
• Create/modify/delete small dictionary 100,000 times
• Create/modify/delete large dictionary 100 times
• Create custom class and instantiate it 100,000 times
• Create Callable and call it 100,000 times Add various GDScript, core, GUI and physics benchmarks #32
• Use load() on resource 100,000 times
• Create 100,000 Colors from HTML strings
• Go through long if 100,000 times
• Go through long match 100,000 times (with same logic as if)
• Go through heavily nested if 100,000 times
• Go through heavily nested match 100,000 times
• Long for loop
• Long while loop (with same logic as for)
• String formatting (%, no-op) Add various GDScript, core, GUI and physics benchmarks #32
• String formatting (%, simple) Add various GDScript, core, GUI and physics benchmarks #32
• String formatting (%, complex) Add various GDScript, core, GUI and physics benchmarks #32
• String formatting (String.format(), no-op) Add various GDScript, core, GUI and physics benchmarks #32
• String formatting (String.format(), simple) Add various GDScript, core, GUI and physics benchmarks #32
• String formatting (String.format(), complex) Add various GDScript, core, GUI and physics benchmarks #32
• var2str() on all core types 10,000 times
• str2var() on all core types 10,000 times
• Find whether a small number is prime (with recursion)
• Find whether a medium number is prime (with recursion)
• Find whether a large number is prime (with recursion)

Navigation

• AStar2D
• AStar3D
• 2D pathfinding
• 3D pathfinding

Physics

All tests should be done in 2D and 3D when possible.
To only represent physics performance, no visual representation should be made unless a debug flag is passed (e.g. --debug-benchmarks).

• 1,000 StaticBodies (box)
• 100 box CharacterBodies (falling on StaticBody box)
• 1,000 box CharacterBodies (falling on StaticBody box)
• 1,000 StaticBodies (complex concave shape)
• 100 box CharacterBodies (falling on StaticBody complex concave shape)
• 1,000 box CharacterBodies (falling on StaticBody complex concave shape)

Repeat with box, sphere, capsule, cylinder, simple convex (quickhull) and complex convex (VHACD):

• 100 RigidDynamicBodies (falling on StaticBody box, placed far apart to avoid collisions with other rigid bodies)
• 100 RigidDynamicBodies (falling on StaticBody box, placed close to each other)
• 1,000 RigidDynamicBodies (falling on StaticBody box, placed far apart to avoid collisions with other rigid bodies)
• 1,000 RigidDynamicBodies (falling on StaticBody box, placed close to each other)
• 100 RigidDynamicBodies (falling on StaticBody complex concave shape, placed far apart to avoid collisions with other rigid bodies)
• 100 RigidDynamicBodies (falling on StaticBody complex concave shape, placed close to each other)
• 1,000 RigidDynamicBodies (falling on StaticBody complex concave shape, placed far apart to avoid collisions with other rigid bodies)
• 1,000 RigidDynamicBodies (falling on StaticBody complex concave shape, placed close to each other)

GUI

• Containers (with lots of nodes)
• RichTextLabel (with lots of lines and no BBCode)
• RichTextLabel (with lots of lines and basic BBCode)
• RichTextLabel (with lots of lines and complex nested BBCode)
• Label (with lots of text) Add various GDScript, core, GUI and physics benchmarks #32

[Mach565]

I think I might have a go at the Sprite2D and Polygon2D benchmarks, just a quick question, though. Should I instance the sprites/polygons through code, or should they already be in the scene when the benchmark is run?

[Calinou]

Should I instance the sprites/polygons through code, or should they already be in the scene when the benchmark is run?

There will be a lot of sprites/polygons to add, so you should instance them through code like the 3D benchmarks are doing.

[Mach565]

Alright, thanks. I'll start work on it now.

[Mach565]

Sprite2D benchmarks should be ready to review, #18

[Mach565]

Polygon2D benchmarks are ready, #19

[tinybitofheaven]

Could I give the DirectionalLight2D and the PointLight2D tasks a try? Additionally, to get around the 2D light limit, should I instantiate multiple background sprites as well as multiple sprites with occluders?

Otherwise, we could end up with lights not rendering.

Alternatively, I could toggle lights on and off, but I'm not sure which method would be preferred.

[Calinou]

Could I give the DirectionalLight2D and the PointLight2D tasks a try?

Sure, feel free to open a pull request 🙂

Additionally, to get around the 2D light limit, should I instantiate multiple background sprites as well as multiple sprites with occluders?

Yes, we need to make sure all lights show up.

[tinybitofheaven]

The PointLight2D and DirectionalLight2D benchmarks are ready for review! #60

[leannelu]

Could I try the GPUParticles2D and CPUParticles2D benchmarks? And are there any specific particle settings for those that I should be setting? Or a specific number of nodes/particles?

[Calinou]

Could I try the GPUParticles2D and CPUParticles2D benchmarks?

Sure, go ahead 🙂

And are there any specific particle settings for those that I should be setting? Or a specific number of nodes/particles?

I'd try creating the particle setups mentioned (that is, one benchmark with lots of particle nodes that have few particles each, and one benchmark with few particle nodes that have lots of particles each). As for the mesh/material, we want to be testing simulation performance as opposed to rendering performance, so I recommend using a small billboarded QuadMesh (0.1 × 0.1 units) with a material that has its shading mode set to Unshaded.

[leannelu]

GPUParticles2D and CPUParticles2D benchmarks are ready to review! #62