Splat shader #12364

keyboardspecialist · 2024-12-06T19:00:52Z

Description

Opening as a draft as some things are still in progress, and this is a long running feature branch.

This adds support for rendering Gaussian splats from 3D Tiles 1.1 using a proposed draft extension here

description in progress

PixelFormat, PixelDatatype - Support added for integer textures with pixel formats R32UI/R32I, RG32UI/RG32I, RGB32UI/RGB32I.

GaussianSplatPipelineStage - This is the old pipeline. May be removed in the future, but doesn't rely on any web assembly.

GaussianSplatTexturePipelineStage - New pipeline. Renders from an attribute texture. Uses web assembly for texture generation and sorting the indexes for retrieving attributes from the texture. Texture format is RGBAUI.

Both use PrimitiveLoadPlan to prep their attributes. The only pipeline simply uses this to keep attributes in CPU memory, so they can sorted later. The new pipeline generates it's texture here, and inserts an index attribute to read from that texture.

GeometryPipelineStage - This is where we determine if we are actually rendering splats. Since we use a point primitive in the glTF we can switch between them at runtime. If disabled, we follow the point cloud pipeline as usual.

Model.js - Sorting updates occur here. We check for camera movement then sort and rebuild the command list.

Model, Cesium3DTileStyle - Styling for scale and toggling splats/point rendering

TODO

GaussianSplatTextureGenerator - Needs to be moved to using TaskProcessor.
Inline documentation

Configuration to use @cesium/wasm-utils once published

Issue number and link

Testing plan

TBD. Will include a sandcastle.

Author checklist

I have submitted a Contributor License Agreement
I have added my name to CONTRIBUTORS.md
I have updated CHANGES.md with a short summary of my change
I have added or updated unit tests to ensure consistent code coverage
I have updated the inline documentation, and included code examples where relevant
I have performed a self-review of my code

interesting scaling decomposed covariance

fragment shader impl

need to get instanced drawarrays working

…ost processing

…loading. Setup for instanced rendering of quads

Attempting to fix splat scaling

new uniforms for camera data, no more computing in vertex shader renamed splat stage define

some clean up

correctly uses viewProjection when sorting

prop cleanup

shader tweaks

Sort commands by OBBs in camera space

ggetz

Thanks for getting this fixed up after the prettier updates @keyboardspecialist! It really helped with the diff.

I notice there are several failing unit tests in CI.
I didn't see any unit tests for the new functionality or files. These will need to be added before the PR can be merged. We typically aim for code coverage of ~90% or more for new code.

ggetz · 2025-01-16T16:13:41Z

temp_wasm/pkg/README.md

@@ -0,0 +1,84 @@
+<div align="center">


To confirm my understanding, all of temp_wasm/ is intended to go away once CesiumGS/cesium-wasm-utils#1 is merged and published, correct?

If so, I think we'll want to add a TODO item for

Configuration to use @cesium/wasm-utils once published

Correct. I added the TODO item to the PR description.

ggetz · 2025-01-16T16:16:02Z

package.json

@@ -52,7 +52,8 @@
  ],
  "dependencies": {
    "@cesium/engine": "^13.1.0",
-    "@cesium/widgets": "^10.1.0"
+    "@cesium/widgets": "^10.1.0",
+    "@cesium/wasm-splats": "file:./temp_wasm/pkg"


The root dependencies here should only include the packages in this repo.

The @cesium/wasm-splats dependency should be moved to @cesium/engine/package.json, since that's the code which depends on the wasm module.

I will fix this when I move to using the published package.

ggetz · 2025-01-16T16:17:35Z

gulpfile.js

@@ -373,6 +373,11 @@ export async function prepare() {
    "Tools/jsdoc/cesium_template/static/styles/prism.css",
  );

+  copyFileSync(


This is nit-picky, but can we move this line up with the other source code dependencies up around line 351? It will help it not get lost among the other dev processes like building documentation.

No problem. I moved it up.

ggetz · 2025-01-16T16:19:07Z

scripts/build.js

@@ -1133,6 +1137,7 @@ export async function buildCesium(options) {
    outbase: "packages/widgets/Source",
  });

+


These appear to be unneeded whitespace changes. Do you anticipate needing to update this file? If not, can we checkout the version in main to avoid the noise?

Unneeded. I merged the version from main.

ggetz · 2025-01-16T16:23:43Z

Apps/Sandcastle/gallery/3D Tiles Gaussian Splatting.html

+    />
+    <meta
+      name="description"
+      content="Use Viewer to start building new applications or easily embed Cesium into existing applications."


This description should be updated to a summary of the example. Any key words here will also be helpful, as they will help this example show up when a user searches.

Something along the lines of "Load a gaussian splat dataset of a cell tower as a 3D tileset." would be great.

Updated to a relevant description

"Load 3D Tiles 1.1 Gaussian Splats dataset of a cell tower"

ggetz · 2025-01-16T18:45:03Z

packages/engine/Source/Shaders/Model/ModelVS.glsl


-        gl_PointSize *= show;
+    #ifdef HAS_CUSTOM_VERTEX_SHADER


We usually indent nested compiler directives for readability.

This indentation was lost at some point, and I don't know why. Re-added.

ggetz · 2025-01-16T18:45:47Z

packages/engine/Source/Workers/gaussianSplatTextureGenerator.js

@@ -0,0 +1,34 @@
+import createTaskProcessorWorker from "./createTaskProcessorWorker.js";
+//import defaultValue from "../Core/defaultValue.js";


Remove unused imports

ggetz · 2025-01-16T18:46:00Z

packages/engine/Source/Workers/gaussianSplatSorter.js

@@ -0,0 +1,54 @@
+import createTaskProcessorWorker from "./createTaskProcessorWorker.js";
+//import defaultValue from "../Core/defaultValue.js";


Remove unused imports

ggetz · 2025-01-16T18:46:29Z

packages/engine/Source/Workers/gaussianSplatSorter.js

+    return radix_sort_gaussians_indexes(
+      primitive.positions,
+      primitive.modelView,
+      2048,


Where does 2048 come from?

It was picked as a maximally compatible texture width.

OK. Perhaps this should be defined as a constant somewhere then to make this a bit more explanatory.

ggetz · 2025-01-16T19:41:50Z

packages/engine/Source/Shaders/Model/GaussianSplatVS.glsl

@@ -0,0 +1,187 @@
+#ifndef HAS_SPLAT_TEXTURE
+
+void calcCov3D(vec3 scale, vec4 rot, out float[6] cov3D)


@lilleyse Should probably be the one to do a pass on the shaders.

updates geometry pipeline stage to use modelutility to find attributes

…g as it's only used on the instanced quad Simplified updateGaussianSplatting in Model.js, camera check caused more issues than it solved. Index sorting is fast enough, and resolves some edge cases where fast rotations with lots of tiles would orphan some and leave them unsorted

ggetz · 2025-01-27T17:50:12Z

Thanks for all the updates here @keyboardspecialist!

What is the current status of this PR and where can we help with reviews?

Is the code ready for another review pass?
What is the status of unit tests?
What does a testing plan look like for us to confirm functionality?
Is the inline documentation up to date?
Has GaussianSplatTextureGenerator been updated as per the TODO list item?

keyboardspecialist added 30 commits June 7, 2024 16:20

initial vertex calcs for splats

0103673

removed early returns. we seem to hit them all the time

ea53962

interesting scaling decomposed covariance

covariance fixes

60022f4

fragment shader impl

hack to switch primitive type points to triangle_strips

ba77e31

refactor, use inverse matrices

8d67853

final point primitive experiment. moving to quads

9aaf292

quad drawing.

8c3aea4

need to get instanced drawarrays working

new splat pipeline files

7a3f472

shader org

4f9c886

switched rendering to not need vertex buffer attr

b235c09

tweaks

df28196

detect KHR_gaussian_splatting extension and set props and prime for p…

bdf4eb5

…ost processing

Buffer rework. inject attribute buffer for screen space quad at gltf …

7a4c864

…loading. Setup for instanced rendering of quads

splats but bad projection

c6b8d25

shader cleanup, simplification

a03abdd

minor tweaks

7d5b09a

tweaks

0620dca

Fixed position and color buffers

fe340a4

Attempting to fix splat scaling

scaling?

e352ca2

added new GAUSSIAN_SPLAT pass

3e74c74

pipeline execution for splats

d5f4069

new uniforms for camera data, no more computing in vertex shader renamed splat stage define

splats sorted via countSort

13dfa78

some clean up

invalidates splat commands when camera moves

31e8d13

correctly uses viewProjection when sorting

gaurded splat vertex setup in builddrawcommand

947d9e0

prop cleanup

command rebuild under control

9862eeb

guard gaussian pipeline stage

0069114

sorting fixes

427d490

shader tweaks

cleanup

fcfc3df

opacity fix

f4657c6

small tweak to re-enable instanceDivisor check on attr 0

2270dae

weegeekps and others added 5 commits January 10, 2025 16:27

Committing the wasm package temporarily

635659e

Add orientedBoundingBox to gsplat commands

edca8a8

Sort commands by OBBs in camera space

typo on final sort calc

a327c88

Merge branch 'main' into splat-shader

b0bde10

add weighted distance from center to sort

ef2ec34

ggetz requested changes Jan 16, 2025

View reviewed changes

keyboardspecialist added 23 commits January 17, 2025 14:13

Formatting

e749c92

move copyFileSync for gaussian splat utilities

afcc22a

merge scripts/build.js from main

78ea71a

revert unneeded default value change

36fd99e

remove unneeded error checking

0fa8fe4

dead code cleanup

703da40

dead code removal

ede218f

descriptive naming

40c5271

remove unused imports

95fa1ff

legibility

fbf6e82

added TILE_GLTF check for gaussian splatting validity check

b6af598

logic cleanup on enableShowGaussianSplatting

1c52cd0

scratch matrices for updateGaussianSplatting

0bb27f2

use defined()

84d1890

remove superfluous ?? false

3e0e4cc

remove unused imports

76364db

remove unused imports

dbf21c2

update description

76ea60a

reverting scratch matrices. Async sorting clobbers them across tiles

f98cbbf

switch to VertexAttributeSemantics

be481c3

texture width constant

8183132

added splat indexes to vertex attributes

933c3e3

updates geometry pipeline stage to use modelutility to find attributes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Splat shader #12364

Splat shader #12364

keyboardspecialist commented Dec 6, 2024 •

edited by weegeekps

Loading

ggetz left a comment

ggetz Jan 16, 2025

weegeekps Jan 17, 2025

ggetz Jan 16, 2025

weegeekps Jan 17, 2025

ggetz Jan 16, 2025

keyboardspecialist Jan 24, 2025

ggetz Jan 16, 2025

keyboardspecialist Jan 17, 2025

ggetz Jan 16, 2025

keyboardspecialist Jan 17, 2025

ggetz Jan 16, 2025

keyboardspecialist Jan 17, 2025

ggetz Jan 16, 2025

keyboardspecialist Jan 17, 2025

ggetz Jan 16, 2025

keyboardspecialist Jan 17, 2025

ggetz Jan 16, 2025

keyboardspecialist Jan 17, 2025

ggetz Jan 21, 2025

ggetz Jan 16, 2025

ggetz commented Jan 27, 2025

		@@ -1133,6 +1137,7 @@ export async function buildCesium(options) {
		outbase: "packages/widgets/Source",
		});

		@@ -0,0 +1,34 @@
		import createTaskProcessorWorker from "./createTaskProcessorWorker.js";
		//import defaultValue from "../Core/defaultValue.js";

		@@ -0,0 +1,54 @@
		import createTaskProcessorWorker from "./createTaskProcessorWorker.js";
		//import defaultValue from "../Core/defaultValue.js";

		@@ -0,0 +1,187 @@
		#ifndef HAS_SPLAT_TEXTURE

		void calcCov3D(vec3 scale, vec4 rot, out float[6] cov3D)

Splat shader #12364

Are you sure you want to change the base?

Splat shader #12364

Conversation

keyboardspecialist commented Dec 6, 2024 • edited by weegeekps Loading

Description

TODO

Issue number and link

Testing plan

Author checklist

ggetz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggetz commented Jan 27, 2025

keyboardspecialist commented Dec 6, 2024 •

edited by weegeekps

Loading