Procedural IBL Enhancement in CesiumJS #12129

ggetz · 2024-08-16T14:18:30Z

Description

This pull request introduces a new system for "procedural" lighting that builds on the Image-Based Lighting (IBL) approach in CesiumJS, aimed at improving the realism and visual quality of the default lighting environment. The changes focus on techniques that create a plausible environment map dynamically based on the current lighting conditions an existing atmospheric scattering code which renders the sky.

Image-Based Lighting Sandcastle

After

Before

Mirror Ball

After

Before

High Altitude Mirror Ball

After

Before

Dynamic Lighting

TODO: More screenshot comparisons

Key Changes

DynamicEnvironmentMapManager: This new component generates lighting value dynamically based on the existing atmospheric calculations. It uses compute command to generate the following when position or sun position has significantly changed.
- Environment Map: generated to match lighting conditions based on a central location and the current sun position.
- Specular Environment Maps: Implements several levels of specular maps, corresponding to various degrees of surface roughness. This allows for more nuanced reflections and highlights that better respond to scene lighting.
- Spherical Harmonic Diffuse Lighting: Computes of spherical harmonic values for diffuse lighting, enhancing the quality and realism of the light diffusion across different surfaces.
The computed values from the new IBL setup are directly used in conjunction with the scene's directional light source, primarily the sun, to establish comprehensive lighting effects across models and 3D Tilesets.
I've chosen to make the DynamicEnvironmentMapManager options accessible via the Model and Cesium3DTileset only, at least for this PR, mainly due to the performance impact if these values are constantly updated.

Additional Updates

Fix Spherical Harmonic reconstruction: as per the filament maintainers
Fix for ImageBasedLighting.imageBasedLightingFactor: This property was broken and went unnoticed.
Removal of ImageBasedLighting.luminanceAtZenith: This property was removed instead of deprecated because there is no direct equivalent in the new implementation, and is likely not widely used (based on the fact that the above bug went unnoticed).
Modifications to CubeMap: New utility functions in support of the operations in DynamicEnvironmentMapManager, such as copying a texture to a specific face.

Issue number and link

N/A

Testing plan

Pull down this branch and run unit tests locally (I had to tweak rendering tests to account for the new lighting. Please double check me here.)
Run the development PBR Lighting sandcastle example and make sure all the sliders/buttons work.
Review all Sandcastle examples and ensure new lighting parameters are adequate for all cases.

Author checklist

…l-ibl

github-actions · 2024-08-16T14:18:51Z

Thank you for the pull request, @ggetz!

✅ We can confirm we have a CLA on file for you.

ggetz · 2024-08-26T17:34:18Z

packages/engine/Source/Renderer/ShaderProgram.js

  if (defined(uniformMap)) {
    const manualUniforms = this._manualUniforms;
    len = manualUniforms.length;
    for (i = 0; i < len; ++i) {
      const mu = manualUniforms[i];
-      mu.value = uniformMap[mu.name]();
+      try {


This update is not strictly necessary, but I found it helpful for debugging.

ggetz · 2024-10-04T13:28:28Z

@jjspace Could you please take a review pass on this?

jjspace · 2024-10-11T17:32:09Z

@ggetz Couple initial comments playing with the new sandcastle.

When I have a small model selected and modify the height the camera zooms really far out. I saw the comment about not handling small models well but is there a way to mitigate that when modifying the height slider?
As mentioned in standup I feel like models are overly blue. Not sure if it's the result of these changes directly or our general visual changes as a whole. For example, this pot at height 400 almost looks like it's glowing blue, like it's emissive itself instead of just reflecting the light around it.

Same with the wicker ball. I'd expect it to be brighter in the areas that are highlighted but I'm not completely sure I'd expect the highlights to be so bluish in color? Maybe I'm just not imagining what the materials would look like correctly
When modifying the height value it doesn't seem that the lighting is updated as I thought you said it should.
Pot loaded at 400 then moved to 80,000ish:

Pot reloaded already at 80,000ish:

jjspace

Had a handful of really small, mostly optional, code comments in addition to the comments above

I don't know if I feel fully qualified to review the shaders but they look ok to me at a cursory glance

jjspace · 2024-10-11T14:45:29Z

packages/engine/Source/Renderer/CubeMap.js

+    for (const faceName of CubeMap.faces()) {
      const face = source[faceName];


This is a bit of a nitpick but I find this naming change confusing.

Given a function named CubeMap.faces() I would expect that to return a list of actual faces that I could iterate over like const face of CubeMap.faces() (which I was going to suggest this be changed to). But the next line here implies that it's not actually the face itself but still only a face name. The type of the iterable also still says they're FaceName values.

jjspace · 2024-10-11T14:48:15Z

packages/engine/Source/Renderer/CubeMap.js

- *
- * @type {Iterable<CubeMap.FaceName>}
- *
+ * @type {iterable<CubeMap.FaceName>}


Suggested change

* @type {iterable<CubeMap.FaceName>}

* @type {Iterable<CubeMap.FaceName>}

jjspace · 2024-10-11T14:53:34Z

packages/engine/Source/Renderer/CubeMapFace.js

+  height,
+  level,
+) {
+  xOffset = defaultValue(xOffset, 0);


Given that we're trying to remove defaultValue I think it's good to avoid adding new calls to avoid future conflicts?

Suggested change

xOffset = defaultValue(xOffset, 0);

xOffset = xOffset ?? 0;

jjspace · 2024-10-11T14:56:10Z

packages/engine/Source/Renderer/CubeMapFace.js

+ * @throws {DeveloperError} framebufferYOffset must be greater than or equal to zero.
+ * @throws {DeveloperError} xOffset + source.width must be less than or equal to width.
+ * @throws {DeveloperError} yOffset + source.height must be less than or equal to height.
+ * @throws {DeveloperError} This CubeMap was destroyed, i.e., destroy() was called.


More of a meta question: All of these errors get thrown only in development because of the pragma.debug. Given that we use the JSDoc to generate our actual documentation but these will never affect downstream users should they actually be documented like this?

jjspace · 2024-10-11T15:41:17Z

packages/engine/Source/Scene/Cesium3DTileset.js

+  if (!this._environmentMapManager.isDestroyed()) {
+    this._environmentMapManager.destroy();
+  }


No change needed but curious about this pattern.

It feels like the responsibility for this check should live in the destroy function itself. I feel like you should be able to just call destroy() regardless and let it handle checking if it's already destroyed and essentially turn into a noop function. Then in this function the logic would just be "We know we want to guarantee this is destroyed so call destroy()" and no wrapper check for if it's already destroyed?

jjspace · 2024-10-11T18:19:10Z

packages/engine/Source/Scene/Model/ImageBasedLightingPipelineStage.js

+      defined(sphericalHarmonicCoefficients) &&
+      defined(sphericalHarmonicCoefficients[0])


It's a slightly weird syntax so completely optional but Optional Chaining does work on array access

Suggested change

defined(sphericalHarmonicCoefficients) &&

defined(sphericalHarmonicCoefficients[0])

defined(sphericalHarmonicCoefficients?.[0])

jjspace · 2024-10-11T18:24:32Z

packages/engine/Source/Scene/Model/ImageBasedLightingPipelineStage.js

+  const sphericalHarmonicCoefficients = defaultValue(
+    imageBasedLighting.sphericalHarmonicCoefficients,
+    environmentMapManager.sphericalHarmonicCoefficients,
+  );


Suggested change

const sphericalHarmonicCoefficients = defaultValue(

imageBasedLighting.sphericalHarmonicCoefficients,

environmentMapManager.sphericalHarmonicCoefficients,

);

const sphericalHarmonicCoefficients =

imageBasedLighting.sphericalHarmonicCoefficients ??

environmentMapManager.sphericalHarmonicCoefficients;

jjspace · 2024-10-11T18:34:09Z

packages/engine/Source/Scene/Model/Model3DTileContent.js

@@ -263,6 +263,11 @@ Model3DTileContent.prototype.update = function (tileset, frameState) {
        : undefined;
  }

+  const tilesetEnvironmentMapManager = tileset.environmentMapManager;
+  if (model.environmentMapManager !== tilesetClippingPlanes) {
+    model._environmentMapManager = tilesetEnvironmentMapManager;


Should this be using the actual setter so all that logic runs correctly?

Suggested change

model._environmentMapManager = tilesetEnvironmentMapManager;

model.environmentMapManager = tilesetEnvironmentMapManager;

jjspace · 2024-10-11T18:35:12Z

packages/engine/Source/Shaders/Builtin/Functions/computeAtmosphereColor.glsl

+ * @name czm_computeAtmosphereColor
+ * @glslFunction
+ *
+ * @param {czm_rat} primaryRay Ray from the origin to sky fragment to in world coords (low precision)


🧀 🐀

Suggested change

* @param {czm_rat} primaryRay Ray from the origin to sky fragment to in world coords (low precision)

* @param {czm_ray} primaryRay Ray from the origin to sky fragment to in world coords (low precision)

jjspace · 2024-10-11T18:40:59Z

packages/engine/Source/Shaders/Model/GeometryStageFS.glsl

@@ -3,7 +3,7 @@ void geometryStage(out ProcessedAttributes attributes)
  attributes.positionMC = v_positionMC;
  attributes.positionEC = v_positionEC;

-  #if defined(COMPUTE_POSITION_WC_CUSTOM_SHADER) || defined(COMPUTE_POSITION_WC_STYLE) || defined(COMPUTE_POSITION_WC_ATMOSPHERE)
+  #if defined(COMPUTE_POSITION_WC_CUSTOM_SHADER) || defined(COMPUTE_POSITION_WC_STYLE) || defined(COMPUTE_POSITION_WC_ATMOSPHERE) 


Suggested change

#if defined(COMPUTE_POSITION_WC_CUSTOM_SHADER) || defined(COMPUTE_POSITION_WC_STYLE) || defined(COMPUTE_POSITION_WC_ATMOSPHERE)

#if defined(COMPUTE_POSITION_WC_CUSTOM_SHADER) || defined(COMPUTE_POSITION_WC_STYLE) || defined(COMPUTE_POSITION_WC_ATMOSPHERE)

ggetz added 17 commits July 17, 2024 10:34

Draft

1c587ce

Merge branch 'main' into procedural-ibl

c0f6f89

Merge remote-tracking branch 'origin/ibl-roughness' into procedural-ibl

d835dd1

Draft

af28f30

Draft

ba94101

Merge branch 'main' into procedural-ibl

c4c734c

Merge remote-tracking branch 'origin/ibl-cubemaps' into procedural-ibl

e303b72

Convolve specular maps

a59ef54

Merge remote-tracking branch 'origin/ibl-multiscatter' into procedura…

7532157

…l-ibl

Draft

3e7e4ec

cleanup

d0a72cd

Merge branch 'main' into procedural-ibl

f0f1cd4

Cleanup, add color modifiers

1e1ef3c

Merge branch 'main' into procedural-ibl

e9ac951

Dynamic lighting

9930587

Hook up 3D Tiles

7458f0a

Cleanup

84bbd77

ggetz added 4 commits August 19, 2024 14:05

Cleanup, fix race condition

53d45a0

Fix sky orientation

b85a22e

Cleanup inline doc

5b4ee24

Shader cleanup

0a1a8c7

ggetz commented Aug 26, 2024

View reviewed changes

ggetz added 5 commits August 26, 2024 13:40

Update CHANGES.md

eb90daa

Minor environment map updates

21045b7

Minor updates, spec fixes

c870ca2

Spec cleanup

092f9b7

Spec updates for environment maps

3f8c1b1

ggetz mentioned this pull request Aug 30, 2024

Allow changing the HDR tonemap #12160

Merged

7 tasks

Draft

298de97

Merge branch 'main' into procedural-ibl

cca3a73

ggetz added the theme - visual quality label Sep 25, 2024

ggetz added 9 commits September 25, 2024 16:10

Merge branch 'main' into procedural-ibl

feae2a9

Performance pass, cleanup

6b2adb1

Performance pass and cleanup

3597350

Merge branch 'main' into procedural-ibl

4a868aa

Prettier

3bb4c37

Update SAndcastle examples

8a96cc2

Unit test fixes

bca4341

Fix model specs

8dcf25d

Fix specs, last SH coefficient

bdc4d15

ggetz marked this pull request as ready for review October 4, 2024 13:28

ggetz added 3 commits October 4, 2024 11:01

Fix import

219ba6d

Specs fixes for --webgl-stub

17690a1

Fix docs, update IBL example

2d6bc61

jjspace reviewed Oct 11, 2024

View reviewed changes

ggetz added 3 commits October 15, 2024 16:51

Merge branch 'main' into procedural-ibl

15ff41c

Account for ground albedo

2b3907c

Update lighting example

abd0e39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Procedural IBL Enhancement in CesiumJS #12129

Procedural IBL Enhancement in CesiumJS #12129

ggetz commented Aug 16, 2024 •

edited

Loading

github-actions bot commented Aug 16, 2024

ggetz Aug 26, 2024 •

edited

Loading

ggetz commented Oct 4, 2024

jjspace commented Oct 11, 2024

jjspace left a comment •

edited

Loading

jjspace Oct 11, 2024

jjspace Oct 11, 2024

jjspace Oct 11, 2024

jjspace Oct 11, 2024

jjspace Oct 11, 2024

jjspace Oct 11, 2024

jjspace Oct 11, 2024

jjspace Oct 11, 2024

jjspace Oct 11, 2024

jjspace Oct 11, 2024

		for (const faceName of CubeMap.faces()) {
		const face = source[faceName];

	* @type {iterable<CubeMap.FaceName>}
	* @type {Iterable<CubeMap.FaceName>}

		defined(sphericalHarmonicCoefficients) &&
		defined(sphericalHarmonicCoefficients[0])

	defined(sphericalHarmonicCoefficients) &&
	defined(sphericalHarmonicCoefficients[0])
	defined(sphericalHarmonicCoefficients?.[0])

	model._environmentMapManager = tilesetEnvironmentMapManager;
	model.environmentMapManager = tilesetEnvironmentMapManager;

	* @param {czm_rat} primaryRay Ray from the origin to sky fragment to in world coords (low precision)
	* @param {czm_ray} primaryRay Ray from the origin to sky fragment to in world coords (low precision)

	#if defined(COMPUTE_POSITION_WC_CUSTOM_SHADER) \|\| defined(COMPUTE_POSITION_WC_STYLE) \|\| defined(COMPUTE_POSITION_WC_ATMOSPHERE)
	#if defined(COMPUTE_POSITION_WC_CUSTOM_SHADER) \|\| defined(COMPUTE_POSITION_WC_STYLE) \|\| defined(COMPUTE_POSITION_WC_ATMOSPHERE)

Procedural IBL Enhancement in CesiumJS #12129

Are you sure you want to change the base?

Procedural IBL Enhancement in CesiumJS #12129

Conversation

ggetz commented Aug 16, 2024 • edited Loading

Description

Dynamic Lighting

Key Changes

Additional Updates

Issue number and link

Testing plan

Author checklist

github-actions bot commented Aug 16, 2024

ggetz Aug 26, 2024 • edited Loading

Choose a reason for hiding this comment

ggetz commented Oct 4, 2024

jjspace commented Oct 11, 2024

jjspace left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggetz commented Aug 16, 2024 •

edited

Loading

ggetz Aug 26, 2024 •

edited

Loading

jjspace left a comment •

edited

Loading