[WIP] VR + Post Processing

[takahirox]

Let me share my WIP PR trying to enable VR + post-processing.
Feedback is welcome because I'm feeling like there'd be better designs.

Video: https://twitter.com/superhoge/status/866687749385891841

Policy:

• keep using EffectComposer + Pass approach
• reusable existing shader pass
• enable different post-processing for each left/right view

Basic idea:

1. Render normally with WebVRCamera
2. Extract the left/right half of the result of 1 to half size renderTarget.
3. Apply pass to each half
4. Combine the left/right results

Disadvantage:

• Not good Performance. Rendering# increases in five. (two for 2., two for 3., one for 4.)

How to use:

// Glitch example
camera = new THREE.WebVRCamera( vrdisplay, renderer );
composer = new THREE.EffectComposer( renderer );
composer.setSize( window.innerWidth, window.innerHeight );
composer.addPass( new THREE.RenderPass( scene, camera ) );
composer.addPass( new THREE.ShaderVRPass( vrdisplay, new THREE.GlitchPass() ) );
composer.passes[ composer.passes.length - 1 ].renderToScreen = true;

Change:

• Add ShaderVRPass which acts as the basic idea I wrote above.
• Update WebGLState properties instead of WebGLRenderer properties for rendering with ArrayCamera(see Not update renderer property for ArrayCamera render #11051)
• Add a way to prevent vrDisplay.submitFrame() invoking from WebVRCamera because I wanna invoke it after post-processing.
• Extract update logic (ex: glitch calculation) from pass.render() to pass.update() method because I wanna call pass.render() two times but run the update logic once in an animation frame.

[wizgrav]

I think that the performance issue in your approach could be solved by having a .renderTarget property on cameras. This property would override the framebuffer per camera similar to the .viewport property of WebGLRendertarget. This way things would get drawn directly in the appropriate target. I'm not sure what the cost of switching framebuffers per object is but I imagine it would amount to much less than an extra full screen pass

[takahirox]

Thanks for the comment.

I need to update this PR because Three.js VR API has been updated.

[takahirox]

@mrdoob

Currently vr.submitFrame(); is called in the end of WebGLRenderer.render() if vr is enabled.

three.js/src/renderers/WebGLRenderer.js

Lines 1238 to 1242 in 146b937

	if ( vr.enabled ) {
	vr.submitFrame();
	}

But we wanna call .submitFrame() after post-processing if post-processing is applied.

There're some options in my mind,

1. add an option to WebGLRenderer or WebVRManager which enables VR but doesn't call .submitFrame()
2. WebGLRenderer has post-processing support (imports EffectComposer?) and calls .submitFrame() after applying post-processing.

Which one do you prefer? Or any better ideas?

(*Effect, like OutlineEffect, would have the same issue)

[wizgrav]

regarding submitFrame, #11582 addresses that the same way VREffect did, by setting an autoSubmitFrame flag

[takahirox]

Thanks!

@mrdoob Will you merge #11582 soon?

[takahirox]

Updated. I've changed the strategy a bit.

EffectComposer manages separating/combining for VR. So no longer we need ShaderVRPass.

Users can reuse *Pass for VR without any changes now.

Video: https://twitter.com/superhoge/status/927923513767337984

	composer = new THREE.EffectComposer( renderer );
	composer.setSize( window.innerWidth, window.innerHeight );

	composer.addPass( new THREE.RenderPass( scene, camera ) );
	composer.addPass( new THREE.UnrealBloomPass( new THREE.Vector2( window.innerWidth, window.innerHeight ), 1.0, 0.7, 0.1 ) );
	composer.addPass( new THREE.ShaderPass( THREE.FilmShader ) );
	composer.addPass( new THREE.GlitchPass() );
	composer.addPass( new THREE.ShaderPass( THREE.FXAAShader ) );

	composer.passes[ composer.passes.length - 1 ].renderToScreen = true;

(I should clean-up/optimize EffectComposer tho!)

[mrdoob]

Are you planning on updating this PR after the scene.onAfterRender() addition?

[takahirox]

Yes, I'm working now 😄

[takahirox]

Feeling like first updating EffectComposer APIs with scene.onAfterRender() even for non-VR. VR support is next.

IMO current EffectComposer isn't fitting to scene.onAfterRender()

[sjlynch]

Any updates on this? This is something I think a lot of us need for our WebVR projects.

I looked into aframe-effects for VR post-processing but it does not seem to be very well maintained. https://github.com/wizgrav/aframe-effects

If three.js could support VR with effectscomposer, it would be killer and IMO is one of the few remaining key features needed to convince people to use Threejs over something like Unity.

[takahirox]

Not yet. I've been busy for WebGL 2.0 support. And I'm slightly leaning toward stereo aware shader code approach as aframe-effects does for the performance....

I'm still think PostProcessing + VR is important so am gonna start working that again soon. Of course if you have similar/better/another ideas, you can make PR.

[arpu]

@takahirox hey any news on this? looks like there are some PRs waiting for this like

#14274

aframe
aframevr/aframe#3645

[takahirox]

I'm revisiting VR + PostProcessing... Came to think Aframe style looks more performant and am gonna try.

[takahirox]

Closing in favor of #15840