Add "merge" blend mode (#6936)

* WIP add new blend mode * WIP add cover blend mode * fix uint24 rendering in merge mode - and add ui to switch rendering modes * remove unused imports * rename to render mode to blend mode * improve in line comments * decide whether a color is valid by alpha < 0. - Fix layer precedence order. * multiple fixes in cover blend mode - fix uint24 rendering - fix rendering layers below if the the upper layer has no data yet - fix color and opacity setting in cover blend mode * fix reading blendmode from backend on inital load * refactoring & make changes ready for review * add changelog entry * add blend mode setting to docs * Apply suggestions from code review Co-authored-by: Philipp Otto <[email protected]> * refactor shader code for blending - make clear that the blend mode only applies to color layers excluding segment layers --------- Co-authored-by: Philipp Otto <[email protected]>
scalableminds · May 25, 2023 · d338889 · d338889
1 parent a7bbbef
commit d338889
Show file tree

Hide file tree

Showing 14 changed files with 125 additions and 18 deletions.
diff --git a/CHANGELOG.unreleased.md b/CHANGELOG.unreleased.md
@@ -12,6 +12,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 
 ### Added
 - Added segment groups so that segments can be organized in a hierarchy (similar to skeletons). [#6966](https://github.com/scalableminds/webknossos/pull/6966)
+- Added a new "cover" blend mode which renders the visible layers on top of each other. The new blend mode can be selected in the Data Rendering settings in the settings tab of the left side bar. [#6936](https://github.com/scalableminds/webknossos/pull/6936)
 - In addition to drag and drop, the selected tree(s) in the Skeleton tab can also be moved into another group by right-clicking the target group and selecting "Move selected tree(s) here". [#7005](https://github.com/scalableminds/webknossos/pull/7005)
 - Added a machine-learning based quick select mode. Activate it via the "AI" button in the toolbar after selecting the quick-select tool. [#7051](https://github.com/scalableminds/webknossos/pull/7051)
 - Added support for remote datasets encoded with [brotli](https://datatracker.ietf.org/doc/html/rfc7932). [#7041](https://github.com/scalableminds/webknossos/pull/7041)

diff --git a/docs/images/blend-mode-example-additive-bosch-et-al.png b/docs/images/blend-mode-example-additive-bosch-et-al.png
diff --git a/docs/images/blend-mode-example-cover-bosch-et-al.png b/docs/images/blend-mode-example-cover-bosch-et-al.png
diff --git a/docs/tracing_ui.md b/docs/tracing_ui.md
@@ -116,6 +116,12 @@ Note, not all control/viewport settings are available in every annotation mode.
 #### Data Rendering
 - `Hardware Utilization`: Adjusts the quality level used for rendering data. Changing this setting influences how many data is downloaded from the server as well as how much pressure is put on the user's graphics card. Tune this value to your network connection and hardware power. After changing the setting, the page has to be refreshed.
 - `Loading Strategy`: You can choose between two different loading strategies. When using "best quality first" it will take a bit longer until you see data, because the highest quality is loaded. Alternatively, "Progressive quality" can be chosen which will improve the quality progressively while loading. As a result, initial data will be visible faster, but it will take more time until the best quality is shown.
+- `Blend Mode`: You can switch between two modes of blending the color layer. The default (Additive) simply sums up all color values of all visible color layers. The cover mode renders all color layers on top of each other. Thus the top most color layer covers the color layers below. This blend mode is especially useful for datasets using multi modality layers. Here is an example for such a dataset published by Bosch et al. [1]:
+
+|Additive Blend Mode        | &nbsp;&nbsp;&nbsp;&nbsp;Cover Blend Mode &nbsp; &nbsp;|
+|:-------------------------:|:-------------------------:|
+|![](./images/blend-mode-example-additive-bosch-et-al.png)|![](./images/blend-mode-example-cover-bosch-et-al.png)|
+
 - `4 Bit`: Toggles data download from the server using only 4 bit instead of 8 bit for each voxel. Use this to reduce the amount of necessary internet bandwidth for WEBKNOSSOS. Useful for showcasing data on the go over cellular networks, e.g 4G.
 - `Interpolation`: When interpolation is enabled, bilinear filtering is applied while rendering pixels between two voxels. As a result, data may look "smoother" (or blurry when being zoomed in very far). Without interpolation, data may look more "crisp" (or pixelated when being zoomed in very far).
 - `Render Missing Data Black`: If a dataset does not contain data at a specific position, WEBKNOSSOS can either render these voxels in "black" or it can try to render data from another magnification.
@@ -138,3 +144,7 @@ The status bar at the bottom of the screen serves three functions:
 1. It shows context-sensitive mouse and keyboard control hints. Depending on your selected annotation tool amd any pressed modifier keys (Shift, CMD, CTRL, ALT, etc) it provides useful interaction hints and shortcuts. 
 2. It provides useful information based on your mouse positioning and which objects it hovers over. This includes the current mouse position in the dataset coordinate space, any segment ID that you hover over, and the currently rendered magnification level (MipMap image pyramid) used for displaying any data.
 3. When working with skeletons, the active node and tree IDs are listed. Use the little pencil icon to select/mark a specific ID as active if required. For more on the active node ID, [see the section on skeleton annotations](./skeleton_annotation.md#nodes_and_trees).
+
+
+##### References
+[1] Bosch, C., Ackels, T., Pacureanu, A. et al. Functional and multiscale 3D structural investigation of brain tissue through correlative in vivo physiology, synchrotron microtomography and volume electron microscopy. Nat Commun 13, 2923 (2022). https://doi.org/10.1038/s41467-022-30199-6
diff --git a/frontend/javascripts/messages.tsx b/frontend/javascripts/messages.tsx
@@ -45,6 +45,7 @@ export const settings: Partial<Record<keyof RecommendedConfiguration, string>> =
   gpuMemoryFactor: "Hardware Utilization",
   overwriteMode: "Volume Annotation Overwrite Mode",
   useLegacyBindings: "Classic Controls",
+  blendMode: "Blend Mode",
   renderWatermark: "Logo in Screenshots",
 };
 export const settingsTooltips: Partial<Record<keyof RecommendedConfiguration, string>> = {
@@ -73,6 +74,8 @@ export const settingsTooltips: Partial<Record<keyof RecommendedConfiguration, st
   mouseRotateValue: "Rotation speed when using the mouse to drag the rotation.",
   zoom: "Zoom in or out in the data viewports",
   displayScalebars: "Show a scale in the lower-right corner of each viewport",
+  blendMode:
+    "Set the blend mode for the dataset. The additive mode (default) adds the data values of all color layers. In cover mode, color layers are rendered on top of each other so that the data values of lower color layers are hidden by values of higher layers.",
   renderWatermark: "Show a WEBKNOSSOS logo in the lower-left corner of each screenshot.",
 };
 export const layerViewConfigurations: Partial<Record<keyof DatasetLayerConfiguration, string>> = {

diff --git a/frontend/javascripts/oxalis/constants.ts b/frontend/javascripts/oxalis/constants.ts
@@ -349,3 +349,8 @@ export enum LOG_LEVELS {
   ERROR = "ERROR",
   CRITICAL = "CRITICAL",
 }
+
+export enum BLEND_MODES {
+  Additive = "Additive",
+  Cover = "Cover",
+}
diff --git a/frontend/javascripts/oxalis/geometries/materials/plane_material_factory.ts b/frontend/javascripts/oxalis/geometries/materials/plane_material_factory.ts
@@ -1,6 +1,6 @@
 import * as THREE from "three";
 import _ from "lodash";
-import type { OrthoView, Vector3 } from "oxalis/constants";
+import { BLEND_MODES, OrthoView, Vector3 } from "oxalis/constants";
 import {
   ViewModeValues,
   OrthoViewValues,
@@ -216,6 +216,7 @@ class PlaneMaterialFactory {
       activeCellIdLow: {
         value: new THREE.Vector4(0, 0, 0, 0),
       },
+      blendMode: { value: 1.0 },
     };
 
     const activeMagIndices = getActiveMagIndicesForLayers(Store.getState());
@@ -572,6 +573,15 @@ class PlaneMaterialFactory {
         true,
       ),
     );
+    this.storePropertyUnsubscribers.push(
+      listenToStoreProperty(
+        (storeState) => storeState.datasetConfiguration.blendMode,
+        (blendMode) => {
+          this.uniforms.blendMode.value = blendMode === BLEND_MODES.Additive ? 1.0 : 0.0;
+        },
+        true,
+      ),
+    );
     this.storePropertyUnsubscribers.push(
       listenToStoreProperty(
         (storeState) => storeState.dataset.dataSource.dataLayers,

diff --git a/frontend/javascripts/oxalis/model/bucket_data_handling/texture_bucket_manager.ts b/frontend/javascripts/oxalis/model/bucket_data_handling/texture_bucket_manager.ts
@@ -60,7 +60,7 @@ function maybePadRgbData(src: Uint8Array | Float32Array, elementClass: ElementCl
     tmpPaddingBuffer[idx++] = src[srcIdx++];
     tmpPaddingBuffer[idx++] = src[srcIdx++];
     tmpPaddingBuffer[idx++] = src[srcIdx++];
-    idx++;
+    tmpPaddingBuffer[idx++] = 255;
   }
 
   return tmpPaddingBuffer;

diff --git a/frontend/javascripts/oxalis/shaders/blending.glsl.ts b/frontend/javascripts/oxalis/shaders/blending.glsl.ts
@@ -0,0 +1,36 @@
+import type { ShaderModule } from "./shader_module_system";
+export const getBlendLayersAdditive: ShaderModule = {
+  code: `
+
+    vec4 blendLayersAdditive(
+      vec4 current_color,
+      vec4 color_to_add
+    ) {
+      return current_color + color_to_add;
+    }
+  `,
+};
+
+export const getBlendLayersCover: ShaderModule = {
+  code: `
+    // Applying alpha blending to merge the layers where the top most layer has priority.
+    // See https://en.wikipedia.org/wiki/Alpha_compositing#Alpha_blending for details.
+    vec4 blendLayersCover(
+      vec4 current_color,
+      vec4 layer_color,
+      bool used_fallback_color
+    ) {
+      float mixed_alpha_factor = (1.0 - current_color.a) * layer_color.a;
+      float mixed_alpha = mixed_alpha_factor + current_color.a;
+      vec3 cover_color_rgb = current_color.a * current_color.rgb + mixed_alpha_factor * layer_color.rgb;
+      // Catching edge case where mixed_alpha is 0.0 and therefore the cover_color would have nan values.
+      float is_mixed_alpha_zero = float(mixed_alpha == 0.0);
+      vec4 cover_color = vec4(cover_color_rgb / (mixed_alpha + is_mixed_alpha_zero), mixed_alpha);
+      cover_color = mix(cover_color, vec4(0.0), is_mixed_alpha_zero);
+      // Do not overwrite current_color if the fallback color has been used.
+      float is_current_color_valid = float(!used_fallback_color);
+      cover_color = mix(current_color, cover_color, is_current_color_valid);
+      return cover_color;
+    }
+  `,
+};
diff --git a/frontend/javascripts/oxalis/shaders/filtering.glsl.ts b/frontend/javascripts/oxalis/shaders/filtering.glsl.ts
@@ -104,7 +104,12 @@ const getMaybeFilteredColor: ShaderModule = {
 export const getMaybeFilteredColorOrFallback: ShaderModule = {
   requirements: [getMaybeFilteredColor],
   code: `
-    vec4 getMaybeFilteredColorOrFallback(
+    struct MaybeFilteredColor {
+      vec4 color;
+      bool used_fallback_color;
+    };
+
+    MaybeFilteredColor getMaybeFilteredColorOrFallback(
       float layerIndex,
       float d_texture_width,
       float packingDegree,
@@ -113,14 +118,16 @@ export const getMaybeFilteredColorOrFallback: ShaderModule = {
       vec4 fallbackColor,
       bool supportsPrecomputedBucketAddress
     ) {
-      vec4 color = getMaybeFilteredColor(layerIndex, d_texture_width, packingDegree, worldPositionUVW, suppressBilinearFiltering, supportsPrecomputedBucketAddress);
+      MaybeFilteredColor maybe_filtered_color;
+      maybe_filtered_color.used_fallback_color = false;
+      maybe_filtered_color.color = getMaybeFilteredColor(layerIndex, d_texture_width, packingDegree, worldPositionUVW, suppressBilinearFiltering, supportsPrecomputedBucketAddress);
 
-      if (color.a < 0.0) {
+      if (maybe_filtered_color.color.a < 0.0) {
         // Render gray for not-yet-existing data
-        color = fallbackColor;
+        maybe_filtered_color.color = fallbackColor;
+        maybe_filtered_color.used_fallback_color = true;
       }
-
-      return color;
+      return maybe_filtered_color;
     }
 
     vec4[2] getSegmentIdOrFallback(

diff --git a/frontend/javascripts/oxalis/shaders/main_data_shaders.glsl.ts b/frontend/javascripts/oxalis/shaders/main_data_shaders.glsl.ts
@@ -20,6 +20,7 @@ import compileShader from "./shader_module_system";
 import Constants from "oxalis/constants";
 import { PLANE_SUBDIVISION } from "oxalis/geometries/plane";
 import { MAX_ZOOM_STEP_DIFF } from "oxalis/model/bucket_data_handling/loading_strategy_logic";
+import { getBlendLayersAdditive, getBlendLayersCover } from "./blending.glsl";
 
 type Params = {
   globalLayerCount: number;
@@ -106,6 +107,7 @@ uniform vec3 globalPosition;
 uniform vec3 activeSegmentPosition;
 uniform float zoomValue;
 uniform bool useBilinearFiltering;
+uniform float blendMode;
 uniform vec3 globalMousePosition;
 uniform bool isMouseInCanvas;
 uniform float brushSizeInPixel;
@@ -150,6 +152,8 @@ ${compileShader(
   getWorldCoordUVW,
   isOutsideOfBoundingBox,
   getMaybeFilteredColorOrFallback,
+  getBlendLayersAdditive,
+  getBlendLayersCover,
   hasSegmentation ? convertCellIdToRGB : null,
   hasSegmentation ? getBrushOverlay : null,
   hasSegmentation ? getSegmentationId : null,
@@ -169,7 +173,7 @@ void main() {
     gl_FragColor = vec4(bucketPosition, activeMagIdx) / 255.;
     return;
   }
-  vec3 data_color = vec3(0.0);
+  vec4 data_color = vec4(0.0);
 
   <% _.each(segmentationLayerNames, function(segmentationName, layerIndex) { %>
     vec4 <%= segmentationName%>_id_low = vec4(0.);
@@ -193,7 +197,7 @@ void main() {
 
       vec3 transformedCoordUVW = transDim((<%= name %>_transform * vec4(transDim(worldCoordUVW), 1.0)).xyz);
       if (!isOutsideOfBoundingBox(transformedCoordUVW)) {
-        color_value =
+        MaybeFilteredColor maybe_filtered_color =
           getMaybeFilteredColorOrFallback(
             <%= formatNumberAsGLSLFloat(layerIndex) %>,
             <%= name %>_data_texture_width,
@@ -202,8 +206,9 @@ void main() {
             false,
             fallbackGray,
             !<%= name %>_has_transform
-          ).xyz;
-
+          );
+        bool used_fallback = maybe_filtered_color.used_fallback_color;
+        color_value = maybe_filtered_color.color.rgb;
         <% if (textureLayerInfos[name].packingDegree === 2.0) { %>
           // Workaround for 16-bit color layers
           color_value = vec3(color_value.g * 256.0 + color_value.r);
@@ -222,14 +227,23 @@ void main() {
         color_value = abs(color_value - <%= name %>_is_inverted);
         // Catch the case where max == min would causes a NaN value and use black as a fallback color.
         color_value = mix(color_value, vec3(0.0), is_max_and_min_equal);
-        // Multiply with color and alpha for <%= name %>
-        data_color += color_value * <%= name %>_alpha * <%= name %>_color;
+        color_value = color_value * <%= name %>_alpha * <%= name %>_color;
+        // Marking the color as invalid by setting alpha to 0.0 if the fallback color has been used
+        // so the fallback color does not cover other colors.
+        vec4 layer_color = vec4(color_value, used_fallback ? 0.0 : maybe_filtered_color.color.a * <%= name %>_alpha);
+        // Calculating the cover color for the current layer in case blendMode == 1.0.
+        vec4 additive_color = blendLayersAdditive(data_color, layer_color);
+        // Calculating the cover color for the current layer in case blendMode == 0.0.
+        vec4 cover_color = blendLayersCover(data_color, layer_color, used_fallback);
+        // Choose color depending on blendMode.
+        data_color = mix(cover_color, additive_color, float(blendMode == 1.0));
       }
     }
   <% }) %>
   data_color = clamp(data_color, 0.0, 1.0);
+  data_color.a = 1.0;
 
-  gl_FragColor = vec4(data_color, 1.0);
+  gl_FragColor = data_color;
 
   <% if (hasSegmentation) { %>
   <% _.each(segmentationLayerNames, function(segmentationName, layerIndex) { %>
@@ -246,7 +260,7 @@ void main() {
       float hoverAlphaIncrement = isHoveredCell && <%= segmentationName%>_alpha > 0.0 ? 0.2 : 0.0;
       float proofreadingAlphaIncrement = isActiveCell && isProofreading && <%= segmentationName%>_alpha > 0.0 ? 0.4 : 0.0;
       gl_FragColor = vec4(mix(
-        data_color,
+        data_color.rgb,
         convertCellIdToRGB(<%= segmentationName%>_id_high, <%= segmentationName%>_id_low),
         <%= segmentationName%>_alpha + max(hoverAlphaIncrement, proofreadingAlphaIncrement)
       ), 1.0);

diff --git a/frontend/javascripts/oxalis/store.ts b/frontend/javascripts/oxalis/store.ts
@@ -43,7 +43,7 @@ import type {
   OrthoViewWithoutTD,
   InterpolationMode,
 } from "oxalis/constants";
-import { ControlModeEnum } from "oxalis/constants";
+import { BLEND_MODES, ControlModeEnum } from "oxalis/constants";
 import type { Matrix4x4 } from "libs/mjs";
 import type { SkeletonTracingStats } from "oxalis/model/accessors/skeletontracing_accessor";
 import type { UpdateAction } from "oxalis/model/sagas/update_actions";
@@ -293,6 +293,7 @@ export type DatasetConfiguration = {
   readonly renderMissingDataBlack: boolean;
   readonly loadingStrategy: LoadingStrategy;
   readonly segmentationPatternOpacity: number;
+  readonly blendMode: BLEND_MODES;
 };
 export type PartialDatasetConfiguration = Partial<
   DatasetConfiguration & {

diff --git a/frontend/javascripts/oxalis/view/left-border-tabs/controls_and_rendering_settings_tab.tsx b/frontend/javascripts/oxalis/view/left-border-tabs/controls_and_rendering_settings_tab.tsx
@@ -22,7 +22,7 @@ import { setZoomStepAction } from "oxalis/model/actions/flycam_actions";
 import messages, { settingsTooltips, settings as settingsLabels } from "messages";
 import { userSettings } from "types/schemas/user_settings.schema";
 import type { ViewMode } from "oxalis/constants";
-import Constants from "oxalis/constants";
+import Constants, { BLEND_MODES } from "oxalis/constants";
 import { api } from "oxalis/singletons";
 import Toast from "libs/toast";
 import { ExclamationCircleOutlined } from "@ant-design/icons";
@@ -311,6 +311,21 @@ class ControlsAndRenderingSettingsTab extends PureComponent<ControlsAndRendering
               },
             ]}
           />
+          <DropdownSetting
+            label={<Tooltip title={settingsTooltips.blendMode}>{settingsLabels.blendMode}</Tooltip>}
+            value={this.props.datasetConfiguration.blendMode}
+            onChange={this.onChangeDataset.blendMode}
+            options={[
+              {
+                value: BLEND_MODES.Additive,
+                label: "Additive",
+              },
+              {
+                value: BLEND_MODES.Cover,
+                label: "Cover",
+              },
+            ]}
+          />
           <SwitchSetting
             label={<Tooltip title={settingsTooltips.fourBit}>{settingsLabels.fourBit}</Tooltip>}
             value={this.props.datasetConfiguration.fourBit}

diff --git a/frontend/javascripts/types/schemas/dataset_view_configuration.schema.ts b/frontend/javascripts/types/schemas/dataset_view_configuration.schema.ts
@@ -1,3 +1,4 @@
+import { BLEND_MODES } from "oxalis/constants";
 import { type DatasetLayerConfiguration, type DatasetConfiguration } from "oxalis/store";
 
 export function getDefaultLayerViewConfiguration(
@@ -70,6 +71,7 @@ export const defaultDatasetViewConfigurationWithoutNull: DatasetConfiguration =
   loadingStrategy: "PROGRESSIVE_QUALITY",
   segmentationPatternOpacity: 40,
   layers: {},
+  blendMode: BLEND_MODES.Additive,
 };
 export const defaultDatasetViewConfiguration = {
   ...defaultDatasetViewConfigurationWithoutNull,
@@ -99,6 +101,9 @@ export const baseDatasetViewConfiguration = {
     minimum: 0,
     maximum: 100,
   },
+  blendMode: {
+    enum: Object.values(BLEND_MODES),
+  },
 };
 export const datasetViewConfiguration = {
   ...baseDatasetViewConfiguration,