How does masking with EffectComposer work?

[ViliamKopecky]

I spent awful lot of time with examples/webgl_postprocessing.html but I cannot figure out how to work with masks (EffectComposer, MaskPass, etc.). Maybe I think of them the wrong way - the photoshop way. I'm trying to achieve this:

I would expect usage like this

function init() {
    // ...

    // full render
    renderSceneA = new THREE.RenderPass(sceneA, cameraA);
    renderSceneB = new THREE.RenderPass(sceneB, cameraB);

    // rendering masks
    renderMaskA = new THREE.RenderPass(maskA, maskCameraA);
    renderMaskB = new THREE.RenderPass(maskB, maskCameraB);

    // MaskPass would crop the rendered scene with mask
    // finalComposer would layer up the passes on each other
    finalComposer.addPass(new THREE.MaskPass(renderSceneA, renderMaskA));
    finalComposer.addPass(new THREE.MaskPass(renderSceneB, renderMaskB));

    // ... render to screen etc.
}

function render() {
    finalComposer.render();
}

When I understand this, I would be happy to write some example / tutorial / blogpost about it.

[mrdoob]

@alteredq I'm also curious about this :)

[alteredq]

I would also like to know ;)

I remember this was tricky. I think the main point of confusion may be that the masking is done via stencil buffer (which is like extra black-and-white layer for the render target, with user defined meaning for bits).

You must first render a 3d shape or shapes into stencil buffer. That's the only way how to set the mask, difference between areas with mask on and mask off is whether something was written by the fragment shader (i.e. you can't just render a full screen quad with some picture of the mask, though maybe alphaTest could work).

Then all subsequent rendering into that render target will ignore pixels which don't have stencil bits flipped on. This will last until you clear the mask (= disable stencil test).

For Photoshop / Gimp like masking I guess it would be easier to create ShaderPass using custom shader which would take extra texture parameter with a mask image and then do masking and blending in the fragment shader.

[marklundin]

If it's worth it, I've extended the WebGLRenderer into a new class that adds methods for setting a mask or clearing it for subsequent renders. It just uses the stencil buffer as mentioned, however avoiding using the EffectsComposer allows preservation of the depth buffer which admittedly, might be a bit of an edge case, but I did find a need for it. Also I couldn't get around the antialiasing issue in the renderTargets in the EffectsComposer, but this seems to work.

[ViliamKopecky]

@marklundin Are you going to publish your code? :-)

[marklundin]

Yep, can do in a couple of weeks. Just waiting for a project to launch then able to share. I can add as a separate renderer in the extras directory, or add the methods to the existing WebGLRenderer and submit a pull request.

[zakdances]

Was your code ever published? Just wondering, because EffectComposer and MaskPass are still not covered in the official docs (not that they should be, that's up to mrdoob of course), so examples continue to be useful for some of us still struggling to understand post-processing.

[marklundin]

I'm afraid not. Whilst the idea worked from a technical point of view, my implementation was a little confusing to use and I was worried it might be more detrimental than useful.

Having said that, this seems to be something that crops up a lot, so I'd be happy to contribute something, whether a tutorial, example or some code that would help people construct masks.

[lmcd]

Anyone had any luck getting this to work as per @ViliamKopecky's example?

[sadtaco]

I was able to get it to work, but it required rendering masks to textures and then supplying those textures to a post processing shader's uniform. (main render pass+additional renders+masks)

I don't see a way to do it using alteredq's MaskPass because that requires what you want to have masked be in a separate scene, and it doesn't take into account things from another scene being overtop of it.

What I did is have in my shader a switch to pass to the uniform to tell it to render the object as masked or not(flat black or white), this way things over top in the same scene mask it, and I'm not having to move objects into separate scenes that makes other issues with rendering the scene I really want. That' rendered to a texture that's supplied to a post shader.

It's the same concept as rendering the depth buffer to a texture. Though I certainly wish there was an easier way, that doesn't require multiple scenes like MaskPass.
Ideally, you'd set "maskable" true/false, force override material so everything has a mask shader, and render that, or idealy there'd be a way to set objects as "maskable" true/false and be able to output a stencil buffer based off that, instead of a shader override. I couldn't figure out if that was possible myself.

[tripdragon]

2015 well 2016 now. If no one has done this I'm about to try and will provide an example. But Soooooo hoping someone has one already.

[mrdoob]

The newly added webgl_postprocessing_masking.html should be a easier example to follow.

[tripdragon]

Perfect! Thank you. For others at this time its in the dev branch