Seamless fallback (#3801)

* rewrite shader code so that conversion to relative coords is done as late as possible; remove most of binary fallback rendering code (disables fallback rendering for now)

* tmp: re-add fallback rendering

* make seamless fallback rendering work and re-enable filtering

* fix fallback rendering look up in non xy planes

* fix flow and linting

* properly implement anisotropic fallback rendering and do some clean up

* remove slack reporting for crashed sagas

* improve styling of settings sidebar

* make loading strategy configurable

* further clean up and add comments

* fix flycam test

* fix texture bucket manager test

* fix sketchy null bug

* allow to set loading strategy via preferred task type settings

* improve code comment

* incorporate PR feedback

* properly suppress non-invertible matrices warning

* remove doubling viewport width in flight mode for bucket picking

* update documentation to loading strategy options and improve some other documentation for settings

* update changelog

* add migration for loading strategy

* fix linting

CHANGELOG.md

@@ -10,7 +10,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.md).
 [Commits](https://github.com/scalableminds/webknossos/compare/19.03.0...HEAD)
 
 ### Added
-- 
+- The dataset settings within the tracing view allow to select between different loading strategies now ("best quality first" and "progressive quality"). Additionally, the rendering can use different magnifications as a fallback (instead of only one magnification). [#3801](https://github.com/scalableminds/webknossos/pull/3801)
 
 ### Changed
 - 

MIGRATIONS.md

@@ -5,7 +5,7 @@ This project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
 User-facing changes are documented in the [changelog](CHANGELOG.md).
 
 ## Unreleased
-- 
+- To ensure that the existing behavior for loading data is preserved ("best quality first" as opposed to the new "progressive quality" default) execute: `update webknossos.user_datasetconfigurations set configuration = configuration || jsonb '{"loadingStrategy":"BEST_QUALITY_FIRST"}'`. See [#3801](https://github.com/scalableminds/webknossos/pull/3801) for additional context.
 
 ### Postgres Evolutions:
 - 

docs/tracing_ui.md

@@ -244,7 +244,7 @@ Not all settings are available in every tracing mode.
 
 - `Move Value (nm/s)`:  A high value will speed up movement through the dataset, e.g. when holding down the spacebar. Vice-versa, a low value will slow down the movement allowing for more precision. This setting is especially useful in `Flight mode`.
 
-- `d/f-Switching`: ¯\\_(ツ)_/¯
+- `d/f-Switching`: If d/f switching is disabled, moving through the dataset with `f` will always go *f*orward by *increasing* the coordinate orthogonal to the current slice. Correspondingly, `d` will backwards by decreasing that coordinate. However, if d/f is enabled, the meaning of "forward" and "backward" will change depending on how you create nodes. For example, when a node is placed at z == 100 and afterwards another node is created at z == 90, z will be *decreased* when going forward.
 
 #### Viewport Options / Flight Options
 - `Zoom`: The zoom factor for viewing the dataset. A low value moves the camera really close to the data, showing many details. A high value, will you show more of the dataset but with fewer details and is great for getting an overview or moving around quickly.
@@ -287,6 +287,8 @@ For multi-layer datasets, each layer can be adjusted separately.
 - `Highlight Hovered Cells`: Toggles whether segmented cells will be highlighted in all viewports when hovering over them with the mouse cursor. Useful for identifying the highlighted cell in across all viewports.
 
 #### Quality
-- `4 Bit`: Toggles data download from the server using only 4 Bit instead of 8 Bit for each pixel. Use this to reduce the amount of necessary internet bandwidth for webKnossos. Useful for showcasing data on the go over cellular networks, e.g 3G. 
 - `Quality`: Adjusts the quality level used for data download from the server. "High" will load the original, unmodified data. "Medium" and "Low" will load a downsampled version of the data layer to reduce network traffic. Use this to reduce the amount of necessary internet bandwidth for webKnossos.
-
+- `Loading Strategy`: You can choose between two different loading strategies. When using "best quality first" it will take a bit longer until you see data, because the highest quality is loaded. Alternatively, "Progressive quality" can be chosen which will improve the quality progressively while loading. As a result, initial data will be visible faster, but it will take more time until the best quality is shown.
+- `4 Bit`: Toggles data download from the server using only 4 Bit instead of 8 Bit for each pixel. Use this to reduce the amount of necessary internet bandwidth for webKnossos. Useful for showcasing data on the go over cellular networks, e.g 3G.
+- `Interpolation`: When interpolation is enabled, bilinear filtering is applied while rendering pixels between two voxels. As a result, data may look "smoother" (or blurry when being zoomed in very far). Without interpolation, data may look more "crisp" (or pixelated when being zomed in very far).
+- `Render Missing Data Black`: If a dataset doesn't contain data at a specific position, webKnossos can either render "black" at that position or it can try to render data from another magnification.

frontend/javascripts/admin/tasktype/recommended_configuration_view.js

@@ -36,6 +36,7 @@ const recommendedConfigByCategory = {
     highlightHoveredCellId: false,
     zoom: 0.8,
     renderMissingDataBlack: false,
+    loadingStrategy: "BEST_QUALITY_FIRST",
   },
   flight: {
     clippingDistanceArbitrary: 60,
@@ -66,6 +67,7 @@ export const settingComments = {
   quality: "0 (high), 1 (medium), 2 (low)",
   clippingDistanceArbitrary: "flight/oblique mode",
   moveValue3d: "flight/oblique mode",
+  loadingStrategy: "BEST_QUALITY_FIRST or PROGRESSIVE_QUALITY",
 };
 
 const errorIcon = (

frontend/javascripts/libs/user_settings.schema.json

@@ -28,6 +28,7 @@
         "fourBit": { "type": "boolean" },
         "interpolation": { "type": "boolean" },
         "quality": { "type": "number", "enum": [0, 1, 2] },
+        "loadingStrategy": { "enum": ["BEST_QUALITY_FIRST", "PROGRESSIVE_QUALITY"] },
         "segmentationOpacity": { "type": "number", "minimum": 0, "maximum": 100 },
         "highlightHoveredCellId": { "type": "boolean" },
         "zoom": { "type": "number", "minimum": 0.001 },

frontend/javascripts/messages.js

@@ -28,6 +28,11 @@ export const settings = {
   crosshairSize: "Crosshair Size",
   brushSize: "Brush Size",
   userBoundingBox: "Bounding Box",
+  loadingStrategy: "Loading Strategy",
+  loadingStrategyDescription: `You can choose between loading the best quality first
+    (will take longer until you see data) or alternatively,
+    improving the quality progressively (data will be loaded faster,
+    but it will take more time until the best quality is shown).`,
 };
 
 export default {

frontend/javascripts/oxalis/api/api_latest.js

@@ -653,6 +653,13 @@ class DataApi {
     return segmentationLayer.name;
   }
 
+  /**
+   * Invalidates all downloaded buckets so that they are reloaded on the next movement.
+   */
+  reloadAllBuckets(): void {
+    _.forEach(this.model.dataLayers, dataLayer => dataLayer.cube.collectAllBuckets());
+  }
+
   /**
    * Sets a mapping for a given layer.
    *

frontend/javascripts/oxalis/constants.js

@@ -122,12 +122,9 @@ const VIEWPORT_WIDTH = 376;
 export const ensureSmallerEdge = false;
 
 // Using the following dimensions for the address space,
-// the look up buffer (256**2) is used at a rate of ~ 97%
-// ((32 × 32 × 50 + 16 × 16 × 50) / 256^2 = 0.976563)
-export const addressSpaceDimensions = {
-  normal: [32, 32, 50],
-  fallback: [16, 16, 50],
-};
+// the look up buffer (256**2) is used at a rate of ~ 99%
+// ((36 × 36 × 50) / 256^2 = 0.98877)
+export const addressSpaceDimensions = [36, 36, 50];
 
 export const Unicode = {
   ThinSpace: "\u202f",
@@ -145,8 +142,6 @@ const Constants = {
   MODES_ARBITRARY: ["flight", "oblique"],
   MODES_SKELETON: ["orthogonal", "flight", "oblique"],
 
-  DEFAULT_SEG_ALPHA: 20,
-
   BUCKET_WIDTH: 32,
   BUCKET_SIZE: 32 ** 3,
   VIEWPORT_WIDTH,
@@ -158,7 +153,8 @@ const Constants = {
   MINIMUM_REQUIRED_BUCKET_CAPACITY: 3 * 512,
   LOOK_UP_TEXTURE_WIDTH: 256,
 
-  TDView_MOVE_SPEED: 150,
+  MAX_ZOOM_STEP_DIFF_PREFETCH: 1, // prefetch only fallback buckets for currentZoomStep + 1
+
   MIN_MOVE_VALUE: 30,
   MAX_MOVE_VALUE: 14000,
   MAX_MOVE_VALUE_SLIDER: 1500,
@@ -179,8 +175,6 @@ const Constants = {
   MIN_PARTICLE_SIZE: 1,
   MAX_PARTICLE_SIZE: 20,
 
-  ZOOM_DIFF: 0.1,
-
   DEFAULT_SPHERICAL_CAP_RADIUS: 140,
 
   // !Currently disabled!

frontend/javascripts/oxalis/controller/scene_controller.js

@@ -337,27 +337,30 @@ class SceneController {
     // all buckets necessary for rendering are addressed. The anchorPoint is
     // defined with bucket indices for the coordinate system of the current zoomStep.
     let anchorPoint;
-    // The fallbackAnchorPoint is similar to the anchorPoint, but refers to the
-    // coordinate system of the next zoomStep which is used for fallback rendering.
-    let fallbackAnchorPoint;
 
     const zoomStep = getRequestLogZoomStep(Store.getState());
     for (const dataLayer of Model.getAllLayers()) {
-      [anchorPoint, fallbackAnchorPoint] = dataLayer.layerRenderingManager.updateDataTextures(
-        globalPosition,
-        zoomStep,
-      );
+      anchorPoint = dataLayer.layerRenderingManager.updateDataTextures(globalPosition, zoomStep);
     }
 
     if (optArbitraryPlane) {
-      optArbitraryPlane.updateAnchorPoints(anchorPoint, fallbackAnchorPoint);
+      optArbitraryPlane.updateAnchorPoints(anchorPoint);
       optArbitraryPlane.setPosition(globalPosVec);
     } else {
       for (const currentPlane of _.values(this.planes)) {
-        currentPlane.updateAnchorPoints(anchorPoint, fallbackAnchorPoint);
+        currentPlane.updateAnchorPoints(anchorPoint);
         currentPlane.setPosition(globalPosVec);
         const [scaleX, scaleY] = getPlaneScalingFactor(state, flycam, currentPlane.planeID);
-        currentPlane.setScale(scaleX, scaleY);
+        const isVisible = scaleX > 0 && scaleY > 0;
+        if (isVisible) {
+          this.displayPlane[currentPlane.planeID] = true;
+          currentPlane.setScale(scaleX, scaleY);
+        } else {
+          this.displayPlane[currentPlane.planeID] = false;
+          // Set the scale to non-zero values, since threejs will otherwise
+          // complain about non-invertible matrices.
+          currentPlane.setScale(1, 1);
+        }
       }
     }
   }

frontend/javascripts/oxalis/geometries/arbitrary_plane.js

@@ -52,13 +52,10 @@ class ArbitraryPlane {
     this.materialFactory.stopListening();
   }
 
-  updateAnchorPoints(anchorPoint: ?Vector4, fallbackAnchorPoint: ?Vector4): void {
+  updateAnchorPoints(anchorPoint: ?Vector4): void {
     if (anchorPoint) {
       this.meshes.mainPlane.material.setAnchorPoint(anchorPoint);
     }
-    if (fallbackAnchorPoint) {
-      this.meshes.mainPlane.material.setFallbackAnchorPoint(fallbackAnchorPoint);
-    }
   }
 
   setPosition = ({ x, y, z }: THREE.Vector3) => {

frontend/javascripts/oxalis/geometries/materials/plane_material_factory.js

@@ -113,10 +113,6 @@ class PlaneMaterialFactory {
         type: "v4",
         value: new THREE.Vector3(0, 0, 0),
       },
-      fallbackAnchorPoint: {
-        type: "v4",
-        value: new THREE.Vector3(0, 0, 0),
-      },
       zoomStep: {
         type: "f",
         value: 1,
@@ -187,11 +183,7 @@ class PlaneMaterialFactory {
       },
       addressSpaceDimensions: {
         type: "v3",
-        value: new THREE.Vector3(...addressSpaceDimensions.normal),
-      },
-      addressSpaceDimensionsFallback: {
-        type: "v3",
-        value: new THREE.Vector3(...addressSpaceDimensions.fallback),
+        value: new THREE.Vector3(...addressSpaceDimensions),
       },
     };
 
@@ -290,10 +282,6 @@ class PlaneMaterialFactory {
       this.uniforms.anchorPoint.value.set(x, y, z);
     };
 
-    this.material.setFallbackAnchorPoint = ([x, y, z]) => {
-      this.uniforms.fallbackAnchorPoint.value.set(x, y, z);
-    };
-
     this.material.setSegmentationAlpha = alpha => {
       this.uniforms.alpha.value = alpha / 100;
     };

frontend/javascripts/oxalis/geometries/materials/plane_material_factory_helpers.js

@@ -3,23 +3,22 @@ import * as THREE from "three";
 
 import UpdatableTexture from "libs/UpdatableTexture";
 
+export const channelCountToFormat = {
+  "1": THREE.LuminanceFormat,
+  "2": THREE.LuminanceAlphaFormat,
+  "3": THREE.RGBFormat,
+  "4": THREE.RGBAFormat,
+};
+
 // This function has to be in its own file as non-resolvable cycles are created otherwise
 export function createUpdatableTexture(
   width: number,
   channelCount: number,
   type: THREE.FloatType | THREE.UnsignedByteType | THREE.Uint32BufferAttribute,
   renderer: THREE.WebGLRenderer,
 ): UpdatableTexture {
-  let format;
-  if (channelCount === 1) {
-    format = THREE.LuminanceFormat;
-  } else if (channelCount === 2) {
-    format = THREE.LuminanceAlphaFormat;
-  } else if (channelCount === 3) {
-    format = THREE.RGBFormat;
-  } else if (channelCount === 4) {
-    format = THREE.RGBAFormat;
-  } else {
+  const format = channelCountToFormat[channelCount];
+  if (!format) {
     throw new Error(`Unhandled byte count: ${channelCount}`);
   }
 

frontend/javascripts/oxalis/model/accessors/flycam_accessor.js

@@ -3,7 +3,7 @@ import * as THREE from "three";
 import _ from "lodash";
 import memoizeOne from "memoize-one";
 
-import type { Flycam, OxalisState } from "oxalis/store";
+import type { Flycam, LoadingStrategy, OxalisState } from "oxalis/store";
 import { M4x4, type Matrix4x4 } from "libs/mjs";
 import { ZOOM_STEP_INTERVAL } from "oxalis/model/reducers/flycam_reducer";
 import { clamp, map3 } from "libs/utils";
@@ -18,13 +18,14 @@ import constants, {
   type Vector3,
   type ViewMode,
 } from "oxalis/constants";
-import determineBucketsForOrthogonal from "oxalis/model/bucket_data_handling/bucket_picker_strategies/orthogonal_bucket_picker";
 import determineBucketsForFlight from "oxalis/model/bucket_data_handling/bucket_picker_strategies/flight_bucket_picker";
 import determineBucketsForOblique from "oxalis/model/bucket_data_handling/bucket_picker_strategies/oblique_bucket_picker";
+import determineBucketsForOrthogonal from "oxalis/model/bucket_data_handling/bucket_picker_strategies/orthogonal_bucket_picker";
 import * as scaleInfo from "oxalis/model/scaleinfo";
 
 function calculateTotalBucketCountForZoomLevel(
   viewMode: ViewMode,
+  loadingStrategy: LoadingStrategy,
   datasetScale: Vector3,
   resolutions: Array<Vector3>,
   logZoomStep: number,
@@ -39,7 +40,6 @@ function calculateTotalBucketCountForZoomLevel(
   // Define dummy values
   const position = [0, 0, 0];
   const anchorPoint = [0, 0, 0, 0];
-  const fallbackAnchorPoint = [0, 0, 0, 0];
   const subBucketLocality = [1, 1, 1];
   const sphericalCapRadius = constants.DEFAULT_SPHERICAL_CAP_RADIUS;
 
@@ -76,9 +76,9 @@ function calculateTotalBucketCountForZoomLevel(
     determineBucketsForOrthogonal(
       resolutions,
       enqueueFunction,
+      loadingStrategy,
       logZoomStep,
       anchorPoint,
-      fallbackAnchorPoint,
       areas,
       subBucketLocality,
       abortLimit,
@@ -101,6 +101,7 @@ function calculateTotalBucketCountForZoomLevel(
 // This function is only exported for testing purposes
 export function _getMaximumZoomForAllResolutions(
   viewMode: ViewMode,
+  loadingStrategy: LoadingStrategy,
   datasetScale: Vector3,
   resolutions: Array<Vector3>,
   viewportRects: OrthoViewRects,
@@ -129,6 +130,7 @@ export function _getMaximumZoomForAllResolutions(
     const nextZoomValue = maxZoomValue * ZOOM_STEP_INTERVAL;
     const nextCapacity = calculateTotalBucketCountForZoomLevel(
       viewMode,
+      loadingStrategy,
       datasetScale,
       resolutions,
       currentResolutionIndex,
@@ -196,6 +198,7 @@ export function getRequestLogZoomStep(state: OxalisState): number {
   const { viewMode } = state.temporaryConfiguration;
   const maximumZoomSteps = getMaximumZoomForAllResolutions(
     viewMode,
+    state.datasetConfiguration.loadingStrategy,
     state.dataset.dataSource.scale,
     getResolutions(state.dataset),
     getViewportRects(state),
@@ -222,6 +225,7 @@ export function getMaxZoomValue(state: OxalisState): number {
 
   const maximumZoomSteps = getMaximumZoomForAllResolutions(
     viewMode,
+    state.datasetConfiguration.loadingStrategy,
     state.dataset.dataSource.scale,
     getResolutions(state.dataset),
     getViewportRects(state),