Implement variable rate shading in the Vulkan renderer

[Calinou]

Related to #744, #2183 and #3854.

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

The new Vulkan renderer can provide high-end visuals, but shading all pixels at native resolution is expensive on the GPU.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

With variable rate shading, select pixels can be shaded at a lower rate than 1×1, such as 1×2, 2×1 and 2×2. This allows keeping objects' edges sharp while spending less time on shading. By using a smart algorithm to determine which pixels should be shaded at a lower rate, this improves performance without impacting visuals too much. Dark pixels and pixels with lots of motion are prime candidates for being shaded at a lower rate, since they're less visible by the player.

This proposal targets tier 2 VRS, which provides per-pixel granularity and is generally considered more flexible. Tier 1 VRS only provides per draw-call granularity and is not covered by this proposal.

As of writing, tier 2 VRS is only supported on the following GPU architectures:

• NVIDIA Ampere (RTX 3000 series)
• NVIDIA Turing (RTX 1600 and 2000 series)
• AMD RDNA (RX 6000 series)

VRS support is expected to arrive in the next generation of mobile GPUs as well, such as the RDNA2-powered Samsung Exynos SoC coming out this year.

More information on VRS can be found in this article: https://wickedengine.net/2020/09/06/variable-rate-shading-first-impressions/

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

AMD provides a FidelityFX Variable Shading library that can automatically generate a shading density map based on an input image and motion data, but it's only compatible with DirectX 12 Ultimate and HLSL. It is likely possible to port it to Vulkan and GLSL, but it's not an easy task.
It is also possible to generate this density map based on various parameters from the current viewport:

• Depth of field
• Movement speed (for pixels on the viewport's edges)
• Overall brightness
• Depth discontinuities
• …

VRS will also allow for optimizing an implementation of overscan by shading all pixels outside the viewport at a lower rate.

For testing purposes, there should be a way to globally override VRS on a whole viewport. Some system-wide tools make this possible, but not on all platforms. This override can be used for performance comparisons, but also to improve performance on low-end GPUs that support VRS.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, as this is core rendering functionality.

Is there a reason why this should be core and not an add-on in the asset library?

This is core rendering functionality.

[BastiaanOlij]

Ok, some feedback from my perspective as someone who is going to be working on implementing some of this over the next few weeks. For XR VRS is a very important tool for optimising performance as only about 1/4th of the fragments we render actually appear on screen. This due to the lens distortion required resulting in the need to render at higher resolutions and the image getting condensed the further out the image is. Especially when used in conjunction with eye tracking applying VRS can make an immense difference in performance.

Just like Calinou my interest lies with VRS tier 2 which will have my focus as well. I did want to add to the discussion that VRS tier 1 also has a place within Godot.

A possible use case for VRS tier 1 is optimising rendering the background sky where on lower performing hardware we currently render the background to a half or quarter resolution buffer and then upscale it, VRS tier 1 allows us to do this directly without the inbetween step further benefiting from only rendering visible fragments as sky is rendered after other opaque objects are rendered.

For VRS tier 2 the solution can be neatly divided into two parts.

The first being the obvious application of VRS to the viewport. This involves adding the ability to provide a density texture to the renderer for usage and then supplying this texture as the density attachment for the framebuffer we're using when rendering.

The second part being how we construct the density map and this can be anything from a static map provided by the user, to a map we calculate based on what we're rendering. Obviously for XR we'll need a mode that obtains it from the XR system.

Further note the suggestions made by Calinou are not all practical to use out of the box. The big problem we have is that some are based on the end result of the render which means we need to use the previous frames result to optimise the next frame. That doesn't work for every game type unless we do some form of reprojection or assume the camera isn't moving.

That said, some are nobrainers.

Generation of a density map based on the depth buffer could be sandwiched between the depth pre-pass and the actual rendering of the image and double as a poor mans DOF effect.
Some with generation of a density map based on motion vectors, while Godot doesn't have a motion vector implementation yet we will need to add one in due time for amongst others XR purposes and one possible approach is to do this as a pass right after the depth pre-pass.

Ergo, I envision a new dropdown on the viewport for density maps with options such as:

• Off
• Texture
• XR
• Depth
• Motion
  And with each option having additional settings.

If the GPU doesn't support VRS I guess we just ignore the setting and accept things will be slower.