Use Scene.onBeforeRender() and onAfterRender() in WebGLRenderer for tiled forward lighting and VR post proc #11582

wizgrav · 2017-06-22T11:06:03Z

Use Scene.onBeforeRender on WebGLRenderer to provide a way to get the vr cameras after they are updated for the current frame toimplement forward+ shading and expose the vr cameras for post processing.

edankwan · 2017-11-19T09:17:10Z

@wizgrav can you update and fix the conflict? @mrdoob it should be a simple update. Please offer us some love for VR post effect 🤗

wizgrav · 2017-11-19T19:22:35Z

@edankwan done, it was a silly conflict with a comment(?) @mrdoob the changes are really minimal and still relevant after all this time. It won't just help with post proc. The updated cameras are also needed to construct the textures needed for forward+ shading, after the cameras update and just before the rendering of objects is performed so onBeforeRender serves two purposes here

takahirox · 2017-11-21T11:24:04Z

I don't have strong preference but renderer.vr.autoSubmitFrame sounds a bit better than renderer.autoSubmitFrame because users can easily speculate .autoSubmitFrame is for VR?

@mrdoob

If you have some concerns for renderer.onBeforeRender() while .autoSubmitFrame seems ok to you, can we get .autoSubmitFrame in first because we need it for PostProcessing on VR?

mrdoob · 2017-11-22T10:06:42Z

Hmm... This feels pretty hacky. Can anyone share a snippet of how the code would be used?

wizgrav · 2017-11-22T11:13:12Z

@mrdoob

renderer.onBeforeRender = function (scene, camera, renderTarget) {
  // Use camera to manually cull lights, generate the per screen tile light lists and store them in texture(s)
};

These textures will then be used by custom materials to implement Forward+ shading as described here Forward-Plus-Renderer. The link mentions depth prepass and compute shaders, but they are not needed. In fact the latest iteration of the technique is called clustered lighting and is even more performant without depth prepass ClusteredShading.pdf

Its an alternate implementation of many lights that doesn't need deffered, and is generally faster since we don't always have mrt support. It has many other advantages as well like transparency support.

The reason onBeforeRender is needed there is because we need to tap on the camera after it updates. In WebGLRenderer.render the camera can also be swapped with the vr ArrayCamera, so by tapping where it does, onBeforeRender will be called with the correct camera that will be used for the scene rendering

takahirox · 2017-11-22T11:35:21Z

Here is the very basic PostProcessing + VR example code.

renderer.vr.enabled = true;
renderer.vr.autoSubmitFrame = false;

scene.add( mesh );

mesh2.material.uniforms.tDiffuse.value = renderTarget.texture;
scene2.add( mesh2 );

function animate() {

    requestAnimationFrame( animate );

    // main rendering to renderTarget
    renderer.render( scene, camera, renderTarget );

    // post-processing
    // renderer.vr.enabled = false;  // some post-processing might require non-VR mode
    renderer.render( scene2, camera2 );
    // renderer.vr.enabled = true;

    renderer.submitFrame();

}

animate();

wizgrav · 2017-11-22T12:35:35Z

@takahirox @mrdoob Also some post proc effects need the camera(s) used to render the original scene. Godrays would be an example of that as they need to calculate the center(s) for the radial blur. onBeforeRender would come in handy here as well

mrdoob · 2017-11-23T11:14:33Z

Hmm... How about something like renderer.vr.onBeforeSubmitFrame = function() {}?

wizgrav · 2017-11-23T14:52:11Z

@mrdoob .onBeforeSubmitFrame() would be right before the submitFrame() call right? In this case the scene will be already rendered by the time it is called so the tiled forward technique cannot be implemented. The tiled light textures need to be created just after the cameras have updated but before the scene objects are rendered because their shaders need the up to date textures to perform lighting.

Another proposal would be onBeforeRender() where it is and an onAfterRender() just before the frame is submitted. This way post proc can be performed in the latter and it will work for both normal or vr mode. This adds another method but takes out the autoSubmitFrame flag and is symmetric.

mrdoob · 2017-11-24T07:04:00Z

Another proposal would be onBeforeRender() where it is and an onAfterRender() just before the frame is submitted. This way post proc can be performed in the latter and it will work for both normal or vr mode.

Hmm, do you mind pointing me to the exact lines in the code where we would be calling these functions?

mrdoob · 2017-11-24T07:05:05Z

Oh wait, you already modified the PR.

mrdoob · 2017-11-24T07:08:12Z

src/renderers/WebGLRenderer.js

+
+		if ( tempHandler ) {
+
+			_this._onBeforeRender = null;


I would put this on your app code instead.
We do similar things when using object.onBeforeRender already:

https://github.com/mrdoob/three.js/blob/dev/examples/js/objects/Reflector.js#L158-L162

wizgrav · 2017-11-24T12:30:52Z

@mrdoob I reverted the code, I guess a warning in the docs should suffice for avoiding the infinite recursion and I also didn't like the tempHandler var in the renderer. In the weekend, I'll get on to making an example for tiled forward lighting and post proc

edankwan · 2017-12-13T15:28:35Z

I personally prefer .autoSubmitFrame approach because of the infinite recursion issue @wizgrav mentioned. In order to bypass this issue, we will need to do some hacky stuff which I don't think it is a good solution.

wizgrav · 2018-01-01T20:24:06Z

Happy new year every one, wishes for a prosperous and creative 2018 :)

I finally found some time to write an example for tiled forward lighting. It's probably the most basic implementation, using tile texels as bitmasks to accomodate 32 lights on screen. It can be easily extended to more, though it would not be best approach for more than a couple hundred. There are other approaches using one more intermediate texture that can accommodate thousands of lights.

I added back the autoSubmitFrame flag as @edankwan suggested. I used the onAfterRender() approach for the demo and the mentioned hack for the infinite recursion is visible and not very pleasing imho. Maybe we could keep both approaches as the code footprint is really tiny. After considering @takahirox suggestion, I decided to leave the flag on the renderer and not the VRManager as the renderer is the one actually submitting the frames so it makes more sense to me that he should carry the toggle.

wizgrav · 2018-01-03T20:35:05Z

And here's the example live https://wizgrav.github.io/three.js/examples/webgl_tiled_forward.html I need to cleanup the build files for this PR. Your thoughts @mrdoob ?

pailhead · 2018-01-05T18:30:44Z

src/renderers/WebGLRenderer.js

@@ -1099,6 +1105,8 @@ function WebGLRenderer( parameters ) {

 		}

+		_this.onBeforeRender( _this, scene, camera, renderTarget );


could this possibly be onAfterAutoUpdate ?

We've already started to "render" so it's more like onBeginRender.

pailhead · 2018-01-05T18:43:18Z

It seems to me that the only thing special about the onBeforeRender call is that it may happen after the scene autoUpdate.

Making this call before you actually call myRenderer.render() seems to guarantee only these two things:

the scenegraph objects will have their matrices up to date, if they were set to auto update
the render camera will have a correct matrix if it does not belong to the scenegraph (no parent)

What I was suggesting before was to have an object based onBeforeUpdate and onAfterUpdate that would be called before and after the matrices are computed. It feels like it would give more meaning to:

if ( scene.autoUpdate === true ) scene.updateMatrixWorld();

I'm not sure if i explained it well, since i didn't explain it well the first time i was suggesting this.

With how it's currently written/named it seems that this would make more sense, while doing the exact same thing?


function actualBeforeRenderCall(){...}

myScene.autoUpdate = false

function myRender(){
   myScene.updateMatrixWorld(true)
   actualBeforeRenderCall()
   myRenderer.render(myScene, myCamera)
}

wizgrav · 2018-01-05T19:36:19Z

@pailhead in the case of vr, the provided camera is replaced by an, internally updated, ArrayCamera, This is the only reason onBeforeRender is needed, It provides a convenient structure for both cases, vr or non. Same goes for onAfterRender. Some post proc effects also need the camera(s)

wizgrav · 2018-01-05T21:51:21Z

@mrdoob changed the location of the hooks to be on the Scene object which supported them already from Object3D. I decided to leave .autoSubmitFrame out as, with this arrangement, the renderer can be reused safely inside onAfterRender @edankwan.

WestLangley · 2018-01-05T22:41:26Z