[compositing] Question: How to properly blend and composite isolated groups

[straker]

https://drafts.fxtf.org/compositing-1/#groups

TL;DR: how should isolated groups be blended and composited together? would it be possible to add explicit algorithms and diagrams showing how to composite and blend multiple isolated groups?

I work on axe-core and we are trying to implement support for mix-blend-mode when calculating an elements background color. We've been using the spec to add all the blending algorithms needed to run this calculation, but we've run into a problem trying to calculate compositing of the background colors.

From what I understand from the spec, without mix-blend-mode (i.e. mix-blend-mode: normal), background colors are composited together from bottom-up order. That is, using the following HTML we would blend element A and B together, then the result of that blend would be blended with element C, giving the result of rgb(223, 112, 96) (following the formulas of simple alpha compositing with blending substituted from https://drafts.fxtf.org/compositing-1/#blending). A visual inspection of this result compared to the browser shows that they are the same.

<div id="A" style="background-color: rgba(255, 255, 255, 1.0);">
  <div id="B" style="background-color: rgba(0, 128, 0, 0.25);">
    <div id="C" style="background-color: rgba(255, 0, 0, 0.5);">Test Element</div>
  </div>
</div>

However, once mix-blend-mode is applied to an element, things are different. From the spec I understand that this should create an isolated group since mix-blend-mode creates a stacking context, however what I'm not clear on from the spec is how to then composite an isolate group with other groups. The section on groups has a bit of explanation, but it is unclear to me what that algorithm looks like.

A compositing group is rendered by first compositing the elements of the group onto the initial backdrop. The result of this is a single element containing color and alpha information. This element is then composited onto the group backdrop. Steps shall be taken to ensure the group backdrop makes only a single contribution to the final composite.

If we take the same HTML as before and add mix-blend-mode: difference to element C, what it sounds like should happen is that there are now two groups: an isolated group containing element C, and the root group containing element A, B, and the isolated group.

<div id="A" style="background-color: rgba(255, 255, 255, 1.0);">
  <div id="B" style="background-color: rgba(0, 128, 0, 0.25);">
    <div id="C" style="background-color: rgba(255, 0, 0, 0.5); mix-blend-mode: difference;">Test Element</div>
  </div>
</div>

Root Group
  - #A
  - #B
  - isolated group
    - #C 

From the paragraph, it sounds like the groups are first composited separately onto their initial backdrop, then again onto the group backdrop.

So we start with the root group (since it is the group backdrop of the isolated group) and composite element A and B onto the root groups initial backdrop of white (so blending white and element A, then the result of that with element B). Since mix-blend-mode: difference is a separable blend mode, we use mix-blend-mode: normal for these blendings. The result would be rgb(191, 233, 191). As the root group has no group backdrop, we stop there.

Next we go to the isolated group. It's initial backdrop is transparent black, so we composite element C onto that using mix-blend-mode: difference. That results in rgb(128, 0, 0, 0.5). Then we blend this color with the group backdrop, which is the composite of the root group, so rgb(191, 233, 191) using mix-blend-mode: normal (from what I gather from effect of group isolation on blending). The final result of the color is rgb(160, 117, 96). However a visual inspection of the browser shows that this is not the case.

I've tried a few different combinations of trying to blend the two different groups, but nothing seems to work out correct. Further confusing to me is when there is just a single element with mix-blend-mode: difference, nothing changes about the color, which to me means that transparent black isn't being applied to the isolated group at all. It's not until I add white as the background of the body or html element does the mix-blend-mode take affect.

<div style="background-color: rgba(204, 204, 204, 0.43); mix-blend-mode: difference;">Test element</div>

So how should these groups blend and composite together? Would it be possible to add explicit algorithms and diagrams showing how to composite and blend multiple isolated groups? That to me would be the most helpful in understanding what should happen. Multiple diagrams showing where the transparent black should be inserted into a stack of colors, how groups blend together, etc.

Thanks for your help.

PS. You can see our code for blending here.

[mstange]

I'm not familiar enough with the spec to quote from it, but here's how your example should behave:

1. Composite A into the root group with mix-blend-mode: normal.
2. Composite B into the root group (which now contains the result of step 1) with mix-blend-mode: normal.
3. Create an intermediate surface for the isolated group for C, which starts out fully transparent.
4. Draw the contents of C into the intermediate surface for the isolated group. In this case, this is just the background color of C. The mix-blend-mode of C is ignored while drawing the contents of C.
5. Composite the intermediate surface of C (which the spec calls "single element containing color and alpha information") into the root group with mix-blend-mode: difference.

Expressed with compositing functions "normal(dest, source)" and "difference(dest, source)", the final color is computed as follows:

difference(normal(normal(root backdrop, A), B), C)

Here, normal just does regular source-over compositing, and difference first adjusts the unpremultiplied source RGB values using the "difference" blend mode equation and then does source-over compositing with that adjusted source color.

Since mix-blend-mode: difference is a separable blend mode, we use mix-blend-mode: normal for these blendings.

Kind of unrelated to the main point of this issue, but this sentence doesn't make sense to me. Separability is about the calculations of a single compositing step - it describes whether the calculations can be done independently for the R, G, and B color components. It doesn't have any impact on what to do for other elements in the group.

[mstange]

PS. You can see our code for blending here.

Your code is giving me the correct result for your example, if used as follows:

const A = new Color(255, 255, 255, 1.0);
const B = new Color(0, 128, 0, 0.25);
const C = new Color(255, 0, 0, 0.5);
const AB = flattenColors(B, A);
const ABC = flattenColors(C, AB, "difference")

I think the one thing it's missing is the final unpremultiplication at the end of flattenColors; simpleAlphaCompositing gives you a premultiplied component but the Color constructor takes unpremultiplied components. So the code will give incorrect results whenever the composited result is non-opaque.

[straker]

Thanks for that explanation. It looks like I misread what separable blend modes was (I though it meant the blend mode was applied separately per composite layer, not per color component of RGB).

So it seems that the mix-blend-mode is only applied when the groups are blended together? I tried to follow that through with adding a color D to the example and tried to then see if that would work:

<div id="A" style="background-color: rgba(255, 255, 255, 1.0);">
  <div id="B" style="background-color: rgba(0, 128, 0, 0.25);">
    <div id="C" style="background-color: rgba(255, 0, 0, 0.5); mix-blend-mode: difference;">
      <div id="D" style="background-color: rgba(0, 0, 255, 0.25);">Test Element</div>
    </div>
  </div>
</div>

Root Group
  - #A
  - #B
  - isolated group
    - #C 
    - #D

rootGroup = normal(normal(root backdrop, A), B)
isolatedGroup = normal(C, D)
difference(rootGroup, isolatedGroup)

That almost worked but wasn't quite the right color. I tried difference with the isolateGroup as well but that didn't seem to work either.

[straker]

I think the one thing it's missing is the final unpremultiplication at the end of flattenColors; simpleAlphaCompositing gives you a premultiplied component but the Color constructor takes unpremultiplied components. So the code will give incorrect results whenever the composited result is non-opaque

Oh, So we need to divide the rgb values by the final alpha value?

[mstange]

Oh, So we need to divide the rgb values by the final alpha value?

That's right.

[straker]

Sweet! Thanks for the help. Doing that makes the four color example work now.

[svgeesus]

I think the one thing it's missing is the final unpremultiplication at the end of flattenColors; simpleAlphaCompositing gives you a premultiplied component but the Color constructor takes unpremultiplied components. So the code will give incorrect results whenever the composited result is non-opaque.

This is defined in CSS Color 4: 13.2. Interpolating with alpha:

When the colors to be interpolated are not fully opaque, they are transformed into premultiplied color values
as follows:

• For rectangular orthogonal color coordinate systems, all component values are multiplied by the alpha value
• For cylindrical polar color coordinate systems, the hue angle is not premultiplied, but the other two axes are premultiplied.

To obtain a color value from a premultipled color value, each component which had been premultiplied is divided by the interpolated alpha value.

[chrishtr]

Looks like all is well then? Closing the issue.