Depth Supervision for Gaussian Splatting

[peasant98]

Is your feature request related to a problem? Please describe.
I would like to implement depth supervision for Gaussian Splatting. It seems pretty straightforward to me (I plan to put out a PR), but I am curious if there would need to be a custom .backward() when I include depth information in the loss. I remember that the original gaussian splatting repo required that, but I am not sure in this case.

[kerrj]

Hi! gradients already propagate through depth when you use rasterize_gaussians with depths passed in as the "rgb" (as splatfacto already does, you would just need to render depth when self.training is true in get_outputs). Any losses through the depth should propagate gradients, but this isn't something we have tested thoroughly. @vye16 would know more about the depth gradients since she implemented them in gsplat.

If you're interested in depth supervision, it would also be great to swap ZoeDepth used here for the newer depth anything model that came out recently, seems higher quality/faster/easier to use :)

[kerrj]

It would also be good to explore both metric losses, i.e. MSE with ground truth depth, and relative losses similar to SparseNeRF, or SparseGS. We've messed with these a bit internally but they don't have much of a noticable effect with well captured scenes, more sparse view scenes might be more noticable.

[peasant98]

Thanks! I'll have a PR out for depth-supervised GS this week. Will compare it to regular GS as a benchmark.

[peasant98]

Draft PR here #3182

I have not fully tested it but I have someone who will be running tests. I used DS-GS in one of my recent works (https://touch-gs.github.io/) and it works well in the sparse view!

[pwais]

could this rather be done as a plugin as in the dn-splatter implementation? https://github.com/maturk/dn-splatter