Implement MSAA with WebGL 2

[ebogo1]

CesiumJS has noticeable aliasing artifacts, especially with thin geometry (see #8228). With multisampled render targets in WebGL 2, MSAA is a good step towards fixing them.

To implement MSAA in CesiumJS, we need to decide which render targets will need to get multisampled to avoid artifacts. One idea was to start with the color framebuffer in GlobeDepth.js. There are also questions about how this would interact with translucency passes, and whether there might need to be additional logic for compute commands to perform MSAA if needed.

I'm opening this issue to have a space to discuss the MSAA roadmap and implementation details for CesiumJS.

[lilleyse]

Before adding MSAA in a production way we should get a bunch of representative sandcastle examples together and compare different AA techniques.

For side by side comparisons we should test:

• MSAA 2x
• MSAA 4x
• MSAA 8x
• FXAA
• SMAA
• SSAA
• None

We already have FXAA as a post process. SSAA can be done by setting viewer.resolutionScale to 2.0 (see Resolution Scaling). SMAA is in the same category of post processing AA techniques as FXAA and it should be quick to make a prototype post process stage. There's a ThreeJS example for it here.

We should also put together a very basic, non-production prototype for MSAA to see what it looks like. This can be hacked into GlobeDepth.js.

By default useBrowserRecommendedResolution is false which means we don't render at native resolution on high DPI displays. We do this for performance reasons since for some phones/laptops it means significantly more fragments being shaded. For testing purposes we should see how the AA techniques look with useBrowserRecommendedResolution true or false.

Once we have all these techniques side by side we should decide whether we go down the path of MSAA. It will take a bit of work to integrate it into CesiumJS. MSAA also has a memory cost for both color textures and depth textures.

Here are some other notes that may be relevant:

• MSAA should be positioned as an experimental option since it requires a WebGL 2 context. CesiumJS supports WebGL 2 but the shader translation isn't well optimized (#797) and we don't test it very frequently. However if we're serious about MSAA we should also start becoming serious about testing WebGL 2 and there should be a WebGL 2 sandcastle that we test as part of the release process.
• We'll need to decide what options to expose to the user. This could be an enum in Scene with options FXAA, MSAA, and NONE to start. There should also be an option to control the number of samples (2x, 4x, 8x, etc)
• Do we need to do anything special for HDR? See https://mynameismjp.wordpress.com/2012/10/24/msaa-overview/.
• Any special considerations for transparent backgrounds? See
  • https://groups.google.com/g/webgl-dev-list/c/tt3edoTGUbM?pli=1
  • https://stackoverflow.com/questions/61024584/webgl2-render-to-texture-with-msaa-anti-aliasing-and-transparent-background
• The pick pass should not use MSAA as it will distort the pick colors (which are really integer IDs packed as RGBA8 texture)
• There are many framebuffers and interactions between them in CesiumJS and if we integrate MSAA we will need to be really careful that we update the right ones at the right times.
• Any considerations for translucent geometry and OIT?

[stefnotch]

I think implementing SMAA would be quite lovely. The currently implemented FXAA does work, but the resulting quality leaves a lot to be desired. SMAA supposedly does a much better, while only having a slightly higher performance cost. It would most certainly make the jaggy lines look nicer.

[verybigzhouhai]

Expect this feature

[ebogo1]

SMAA screenshot dump

no AA, viewer.useBrowserRecommendedResolution = false

SMAA, viewer.useBrowserRecommendedResolution = false

no AA, viewer.useBrowserRecommendedResolution = true

SMAA, viewer.useBrowserRecommendedResolution = true

FXAA, viewer.useBrowserRecommendedResolution = true

no AA, viewer.useBrowserRecommendedResolution = true

SMAA, viewer.useBrowserRecommendedResolution = true

FXAA, viewer.useBrowserRecommendedResolution = true

[lilleyse]

@ebogo1 could you put together a sandcastle where you can switch between a few scenarios (including the two above) and a dropdown that lets you select the AA mode (FXAA, SMAA, None) and a checkbox for useBrowserRecommendedResolution? Even better if you could post an S3 link so anyone can try it out.

I've very interested to see how it looks in action. SMAA seems equal or better than FXAA.

[ebogo1]

SMAA findings

I put together a hacky branch with SMAA shaders from Three.js (see Three's SMAAPass.js and SMAAShader.js). First thoughts are that it's seems much more straightforward than trying to integrate MSAA (plus it works with WebGL 1) and looks comparable but better than FXAA. I've posted a few comparison screenshots - #9900 (comment).

---

As a really high-level overview, SMAA consists of three passes:

1. Calculate edges in the rendered scene texture.
2. Calculate blending weights for these edges using two hardcoded texture files. In the smaa branch these are the png files in Source/Assets/Textures/SMAA/, which came from the data URLs that Three.js loads as uniforms (see these lines in Three's SMAAPass).
3. Use the blending weights to perform AA on the rendered scene texture.

In CesiumJS, this means two PostProcessStages for the edge pass and weights pass, a PostProcessStageComposite to combine them in that order, a PostProcessStage for the blend pass, and a final PostProcessStageComposite to feed the blend weights and render texture into the blend pass. This all happens in PostProcessStageLibrary.createSMAAStage().

A couple related resources:

• the smaa implementation from the authors of this paper
• minimal three.js example to render a texture onto a full-screen quad: threejs-smaa.zip
• Sandcastle to print a data URL containing the rendered scene texture (useful with the three.js example above)
• SMAA branch Sandcastle with FXAA, SMAA, and useBrowserRecommendedResolution checkboxes and a few different scenes (Note: labels are rendered upside-down because of hacky changes I made in Texture.js)

[ebogo1]

MSAA findings

I made another branch for testing MSAA; to make it quicker to integrate, there are a couple hacky changes to use SceneFramebuffer instead of GlobeDepth. Only the color and depth textures are multisampled in this branch and if we move forward with MSAA, there is more planning to do for considerations of which other framebuffers to multisample, interpolation options (i.e. should idTexture use MSAA; should depth use the same interpolation as color...), usage with GlobeDepth, etc.

There are some weird issues that may be a result of the SceneFramebuffer hacks - some intersections between different geometries have unexpected aliasing that looks like it isn't getting multisampled at all. Might somehow be related to the change in Context to disable the depth texture, but we haven't figured it out yet.

---

A brief list of changes in the MSAA branch:

• SceneFramebuffer's framebuffer is split into two framebuffers. The first one has multisampled renderbuffer attachments and is bind-ed as gl.READ_FRAMEBUFFER during the gl.blitFramebuffer call. The second has the original texture attachments and is the gl.DRAW_FRAMEBUFFER during the gl.blitFramebuffer call. A similar double framebuffer + blit setup would need to be added for any framebuffer we want to apply MSAA to.
• In most of Scene.render(), the first of SceneFramebuffer's framebuffers is active. The second, non-multisampled framebuffer is only used after the blit at the end of each frame.
• RenderbufferFormat is made to support the DEPTH24_STENCIL8 format for use with WebGL 2 and the RGBA8 format for consistency with the target texture's format during the gl.blitFramebuffer call.

Here is a Sandcastle from the MSAA branch with the same scenarios as the SMAA Sandcastle and a dropdown to change the number of MSAA samples. This Sandcastle requires WebGL 2.

[ebogo1]

Based on the results from prototyping MSAA and SMAA, we decided to move forward with MSAA as an option for WebGL 2 contexts. The plan is split across a couple PRs, but details might change once I work through more of these changes -

1. Remove unused code from Scene.js and GlobeDepth.js - i.e. debugShowGlobeDepth and debugShowPickDepth are broken #6763 still takes up a few lines and the Primitive Framebuffer in GlobeDepth is unused.
2. Create a new wrapper around Framebuffer to manage resources like Renderbuffers and Texture attachments. Refactor GlobeDepth, Sceneframebuffer, OIT, etc. to use this wrapper instead of creating and destroying Framebuffers and attachments directly. The idea is to make it easier to add a multisample = true option for classes using Framebuffers without having to add the same multisample behavior in different places.
3. Add a multisample option to the new Framebuffer wrapper and add MSAA. Two big challenges here are -
   1. Managing the temp and copy depth framebuffers used by GlobeDepth. Depending on how blit interacts with transparent textures and stencil textures, there can definitely be a cleanest way to handle GlobeDepth MSAA.
   2. Finalizing decisions on which framebuffers need to be multisampled when MSAA is enabled and which interpolation each should use if multisampled.
4. (Follow-up) Once MSAA is working, it's possible that it'll expose more code in Scene, View, and classes that own framebuffers that can be refactored.

[FabianAhammer]

Really cool changes coming up, thanks a lot dev team :)

SMAA isn´t going to make it to cesium then, right?

[ebogo1]

@FabianAhammer We're looking at adding SMAA as a toggle-able option like FXAA, but for now focusing on MSAA.