Merge branch 'master' of github.com:scalableminds/webknossos into fea…

…ture-highlight * 'master' of github.com:scalableminds/webknossos: Hide unreported datasets (#3883) Update puppeteer and refresh screenshots (#3914) only show team names of own organization (#3928) Enable merger mode in skeleton and hybrid tracings (#3619) allow uploading nml for public dataset of different orga (#3929) Always make wheel listeners not passive to allow preventDefault (#3939) Enhance tree search functionallity (#3878) add webknossos-connect to setup (#3913) Update README.md (#3923) Add shortcut to maximize golden layout panes (#3927) Perform bucket picking in web workers and other performance optimizations (#3902) remove alt text for abstract brain loading image (#3930) updated documentation front page (#3917)
scalableminds · Mar 25, 2019 · 5b3acc1 · 5b3acc1
2 parents 5278883 + e37934e
commit 5b3acc1
Show file tree

Hide file tree

Showing 90 changed files with 2,453 additions and 2,269 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -257,12 +257,8 @@ jobs:
       - run:
           name: Run screenshot-tests
           command: |
-            for i in {1..5}; do # retry
-              URL=https://master.webknossos.xyz/ \
-              yarn test-screenshot && s=0 && break || s=$?
-              sleep 30
-            done
-            (exit $s)
+            URL=https://master.webknossos.xyz/ \
+            yarn test-screenshot
 
       - store_artifacts:
           path: frontend/javascripts/test/screenshots

diff --git a/.eslintrc.json b/.eslintrc.json
@@ -82,7 +82,16 @@
     "flow-header/flow-header": "error",
     "react/sort-comp": [
       "error",
-      { "order": ["type-annotations", "static-methods", "lifecycle", "everything-else", "render"] }
+      {
+        "order": [
+          "type-annotations",
+          "instance-variables",
+          "static-methods",
+          "lifecycle",
+          "everything-else",
+          "render"
+        ]
+      }
     ]
   }
 }
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -16,13 +16,18 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.md).
 - Neuroglancer precomputed datasets can now be added to webKnossos using the webknossos-connect (wk-connect) service. To setup a wk-connect datastore follow the instructions in the [Readme](https://github.com/scalableminds/webknossos-connect). Afterwards, datasets can be added through "Add Dataset" - "Add Dataset via wk-connect". [#3843](https://github.com/scalableminds/webknossos/pull/3843)
 - The dataset settings within the tracing view allow to select between different loading strategies now ("best quality first" and "progressive quality"). Additionally, the rendering can use different magnifications as a fallback (instead of only one magnification). [#3801](https://github.com/scalableminds/webknossos/pull/3801)
 - The mapping selection dropbown is now sorted alphabetically. [#3864](https://github.com/scalableminds/webknossos/pull/3864)
+- Added the possibility to filter datasets in the dashboard according to their availability. By default, datasets which are missing on disk (e.g., when the datastore was deleted) are not shown anymore. This behavior can be configured via the settings icon next to the search box in the dashboard. [#3883](https://github.com/scalableminds/webknossos/pull/3883)
+- Added merger mode for skeleton and hybrid tracings. It allows to merge segments from e.g. generated segmentations. [#3619](https://github.com/scalableminds/webknossos/pull/3619)
 - The HTML template now includes SEO tags for demo instances and hides internal instances from search engines.
-- A maximize-button was added to the viewports in the annotation view. [#3876](https://github.com/scalableminds/webknossos/pull/3876)
+- A maximize-button was added to the viewports in the annotation view. Maximization can also be toggled with the `.` shortcut. [#3876](https://github.com/scalableminds/webknossos/pull/3876)
+- [webknossos-connect](https://github.com/scalableminds/webknossos-connect) now starts with webKnossos on local and development instances by default. [#3913](https://github.com/scalableminds/webknossos/pull/3913)
 
 ### Changed
 - Improved the flight mode performance for tracings with very large trees (>80.000 nodes). [#3880](https://github.com/scalableminds/webknossos/pull/3880)
 - Tweaked the highlighting of the active node. The inner node looks exactly as a non-active node and is not round, anymore. An active node is circled by a "halo". In arbitrary mode, the halo is hidden and the active node is round. [#3868](https://github.com/scalableminds/webknossos/pull/3868)
+- Improved the performance of moving through a dataset which should make the overall interaction smoother. [#3902](https://github.com/scalableminds/webknossos/pull/3902)
 - Brush size is independent of zoom value, now. This change simplifies volume annotations, as brush sizes can be adapted to certain structures (e.g., vesicles) and don't need to be changed when zooming. [#3868](https://github.com/scalableminds/webknossos/pull/3889)
+- Reworked the search in the trees tab. [#3878](https://github.com/scalableminds/webknossos/pull/3878)
 
 ### Fixed
 - Fixed a bug where failed large save requests lead to inconsistent tracings on the server. [#3829](https://github.com/scalableminds/webknossos/pull/3829)
@@ -31,6 +36,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.md).
 - Fixed interpolation along z-axis. [#3888](https://github.com/scalableminds/webknossos/pull/3888)
 - Fixed that the halo of the active node could cover other nodes. [#3919](https://github.com/scalableminds/webknossos/pull/3919)
 - Fixed that the 3D viewport was partially occluded due to clipping distance issues. [#3919](https://github.com/scalableminds/webknossos/pull/3919)
+- Fixed that scrolling with the mouse wheel over a data viewport also scrolled the page. This bug appeared with the new Chrome version 73. [#3939](https://github.com/scalableminds/webknossos/pull/3939)
 
 ### Removed
 - Removed FPS meter in Annotation View. [#3916](https://github.com/scalableminds/webknossos/pull/3916)

diff --git a/README.md b/README.md
@@ -1,15 +1,13 @@
 # webKnossos
-Cellular-resolution connectomics is currently substantially limited by the throughput and efficiency of data analysis.
-Current solutions require an efficient integration of automated image analysis with massive manual data annotation.
-To scale such annotation efforts it is decisive to be able to crowd source data analysis online.
-Here we present **webKnossos**.
+<img align="right" src="https://static.webknossos.org/images/oxalis.svg" alt="webKnossos Logo" />
+webKnossos is an open-source tool for annotating and exploring large 3D image datasets.
 
-> Boergens, Berning, Bocklisch, Bräunlein, Drawitsch, Frohnhofen, Herold, Otto, Rzepka, Werkmeister, Werner, Wiese, Wissler and Helmstaedter
-webKnossos: efficient online 3D data annotation for connectomics.
-[Nature Methods (2017) DOI:10.1038/NMETH.4331.](https://www.nature.com/articles/nmeth.4331)
-
-![webKnossos logo](https://webknossos.org/images/oxalis.svg)
+* Fly through your data for fast skeletonization and proof-reading
+* Create 3D training data for automated segmentations efficiently
+* Scale data reconstruction projects with crowdsourcing workflows
+* Share datasets and annotations with collaborating scientists
 
+[Start using webKnossos](https://webknossos.org) - [User Documentation](https://docs.webknossos.org) - [Contact us](mailto:[email protected])
 
 [![](	https://img.shields.io/circleci/project/github/scalableminds/webknossos/master.svg?logo=circleci)](https://circleci.com/gh/scalableminds/webknossos)
 [![](https://img.shields.io/github/release/scalableminds/webknossos.svg)](https://github.com/scalableminds/webknossos/releases/latest)
@@ -28,16 +26,23 @@ webKnossos: efficient online 3D data annotation for connectomics.
 * User and task management for high-throughput crowdsourcing
 * Sharing and collaboration features
 * [Standalone datastore component](https://github.com/scalableminds/webknossos/tree/master/webknossos-datastore) for flexible deployments
-* [Supported dataset formats: WKW (Optimized), KNOSSOS cubes](https://github.com/scalableminds/webknossos/wiki/Datasets), [Neuroglancer Precomputed, and BossDB](https://github.com/scalableminds/webknossos-connect)
+* [Supported dataset formats: WKW, KNOSSOS cubes](https://github.com/scalableminds/webknossos/wiki/Datasets), [Neuroglancer Precomputed, and BossDB](https://github.com/scalableminds/webknossos-connect)
 * Supported image formats: Grayscale, Segmentation Maps, RGB, Multi-Channel
 * [Support for 3D mesh rendering and on-the-fly isosurface generation](https://docs.webknossos.org/guides/mesh_visualization)
 * [Documented frontend API for user scripts](https://webknossos.org/assets/docs/frontend-api/index.html), REST API for backend access
 * Open-source development with [automated test suite](https://circleci.com/gh/scalableminds/webknossos)
 * [Docker-based deployment](https://hub.docker.com/r/scalableminds/webknossos/) for production and development
 * [Detailed Documentation](https://docs.webknossos.org)
 
+## Publication
+> Boergens, Berning, Bocklisch, Bräunlein, Drawitsch, Frohnhofen, Herold, Otto, Rzepka, Werkmeister, Werner, Wiese, Wissler and Helmstaedter  
+> webKnossos: efficient online 3D data annotation for connectomics.  
+> [Nature Methods (2017) DOI:10.1038/NMETH.4331.](https://www.nature.com/articles/nmeth.4331)
+
+[Read more about the original publication.](https://publication.webknossos.org)
 
-## Development setup
+
+## Development installation
 ### Docker
 This is the fastest way to try webKnossos.
 Docker CE 17+ and Docker Compose 1.18+ is required.
@@ -141,9 +146,7 @@ yarn start
 Will fetch all Scala, Java and node dependencies and run the application on Port 9000.
 Make sure that the PostgreSQL and Redis services are running before you start the application.
 
-## Production setup
-[See wiki](https://github.com/scalableminds/webknossos/wiki/Production-setup) for recommended production setup.
-
+## Upgrades
 For upgrades, please check the [changelog](CHANGELOG.md) & [migration guide](MIGRATIONS.md).
 
 ## Tests

diff --git a/app/controllers/AnnotationIOController.scala b/app/controllers/AnnotationIOController.scala
@@ -125,10 +125,10 @@ class AnnotationIOController @Inject()(nmlWriter: NmlWriter,
           dataSetName <- assertAllOnSameDataSet(skeletonTracings, volumeTracingsWithDataLocations.headOption.map(_._1)) ?~> "nml.file.differentDatasets"
           organizationNameOpt <- assertAllOnSameOrganization(parseSuccesses.flatMap(s => s.organizationName)) ?~> "nml.file.differentDatasets"
           organizationIdOpt <- Fox.runOptional(organizationNameOpt) {
-            organizationDAO.findOneByName(_).map(_._id)
-          } ?~> Messages("dataSet.noAccess", dataSetName) ~> FORBIDDEN
+            organizationDAO.findOneByName(_)(GlobalAccessContext).map(_._id)
+          } ?~> Messages("dataSet.notFound", dataSetName) ~> FORBIDDEN
           organizationId <- Fox.fillOption(organizationIdOpt) {
-            dataSetDAO.getOrganizationForDataSet(dataSetName)
+            dataSetDAO.getOrganizationForDataSet(dataSetName)(GlobalAccessContext)
           } ?~> Messages("dataSet.noAccess", dataSetName) ~> FORBIDDEN
           dataSet <- dataSetDAO.findOneByNameAndOrganization(dataSetName, organizationId) ?~> Messages(
             "dataSet.noAccess",

diff --git a/app/controllers/DataSetController.scala b/app/controllers/DataSetController.scala
@@ -156,7 +156,7 @@ class DataSetController @Inject()(userService: UserService,
 
   def accessList(organizationName: String, dataSetName: String) = sil.SecuredAction.async { implicit request =>
     for {
-      dataSet <- dataSetDAO.findOneByNameAndOrganization(dataSetName, request.identity._organization) ?~> Messages(
+      dataSet <- dataSetDAO.findOneByNameAndOrganizationName(dataSetName, organizationName) ?~> Messages(
         "dataSet.notFound",
         dataSetName) ~> NOT_FOUND
       allowedTeams <- dataSetService.allowedTeamIdsFor(dataSet._id)

diff --git a/app/controllers/InitialDataController.scala b/app/controllers/InitialDataController.scala
@@ -94,6 +94,7 @@ Samplecountry
     for {
       _ <- insertLocalDataStoreIfEnabled
       _ <- insertLocalTracingStoreIfEnabled
+      _ <- insertConnectDataStoreIfEnabled
       _ <- assertInitialDataEnabled
       _ <- assertNoOrganizationsPresent
       _ <- insertOrganization
@@ -211,6 +212,16 @@ Samplecountry
       }
     } else Fox.successful(())
 
+  def insertConnectDataStoreIfEnabled: Fox[Any] =
+    if (conf.Application.insertLocalConnectDatastore) {
+      dataStoreDAO.findOneByName("connect").futureBox.map { maybeStore =>
+        if (maybeStore.isEmpty) {
+          logger.info("inserting connect datastore")
+          dataStoreDAO.insertOne(DataStore("connect", "http://localhost:8000", "secret-key", isConnector = true))
+        }
+      }
+    } else Fox.successful(())
+
   def insertLocalTracingStoreIfEnabled: Fox[Any] =
     if (conf.Tracingstore.enabled) {
       tracingStoreDAO.findOneByName("localhost").futureBox.map { maybeStore =>

diff --git a/app/models/binary/DataSet.scala b/app/models/binary/DataSet.scala
@@ -259,7 +259,10 @@ class DataSetDAO @Inject()(sqlClient: SQLClient,
       _ <- dataSetDataLayerDAO.updateLayers(old._id, source)
     } yield ()
 
-  def deactivateUnreported(names: List[String], organizationId: ObjectId, dataStoreName: String): Fox[Unit] = {
+  def deactivateUnreported(names: List[String],
+                           organizationId: ObjectId,
+                           dataStoreName: String,
+                           unreportedStatus: String): Fox[Unit] = {
     val inclusionPredicate =
       if (names.isEmpty) "true" else s"name not in ${writeStructTupleWithQuotes(names.map(sanitize))}"
     val deleteResolutionsQuery =
@@ -272,7 +275,7 @@ class DataSetDAO @Inject()(sqlClient: SQLClient,
              and #${inclusionPredicate})"""
     val setToUnusableQuery =
       sqlu"""update webknossos.datasets
-             set isUsable = false, status = 'No longer available on datastore.', scale = NULL
+             set isUsable = false, status = $unreportedStatus, scale = NULL
              where _dataStore = ${dataStoreName} and _organization = ${organizationId}
              and #${inclusionPredicate}"""
     for {

diff --git a/app/models/binary/DataSetService.scala b/app/models/binary/DataSetService.scala
@@ -45,6 +45,8 @@ class DataSetService @Inject()(organizationDAO: OrganizationDAO,
     extends FoxImplicits
     with LazyLogging {
 
+  val unreportedStatus = "No longer available on datastore."
+
   def isProperDataSetName(name: String): Boolean =
     name.matches("[A-Za-z0-9_\\-]*")
 
@@ -171,7 +173,8 @@ class DataSetService @Inject()(organizationDAO: OrganizationDAO,
           case Full(organization) =>
             dataSetDAO.deactivateUnreported(dataSourcesByOrganizationName(organizationName).map(_.id.name),
                                             organization._id,
-                                            dataStoreName)
+                                            dataStoreName,
+                                            unreportedStatus)
           case _ => {
             logger.info(s"Ignoring reported dataset for non-existing organization $organizationName")
             Fox.successful(())
@@ -252,11 +255,16 @@ class DataSetService @Inject()(organizationDAO: OrganizationDAO,
       case _ => Fox.successful(0L)
     }
 
-  def allowedTeamIdsFor(_dataSet: ObjectId)(implicit ctx: DBAccessContext) =
-    dataSetAllowedTeamsDAO.findAllForDataSet(_dataSet)(GlobalAccessContext) ?~> "allowedTeams.notFound"
+  def allowedTeamIdsFor(_dataSet: ObjectId)(implicit ctx: DBAccessContext): Fox[List[ObjectId]] =
+    dataSetAllowedTeamsDAO.findAllForDataSet(_dataSet) ?~> "allowedTeams.notFound"
 
-  def allowedTeamsFor(_dataSet: ObjectId)(implicit ctx: DBAccessContext) =
-    teamDAO.findAllForDataSet(_dataSet)(GlobalAccessContext) ?~> "allowedTeams.notFound"
+  def allowedTeamsFor(_dataSet: ObjectId, requestingUser: Option[User])(
+      implicit ctx: DBAccessContext): Fox[List[Team]] =
+    for {
+      teams <- teamDAO.findAllForDataSet(_dataSet) ?~> "allowedTeams.notFound"
+      // dont leak team names of other organizations
+      teamsFiltered = teams.filter(team => requestingUser.map(_._organization).contains(team._organization))
+    } yield teamsFiltered
 
   def isEditableBy(
       dataSet: DataSet,
@@ -271,22 +279,22 @@ class DataSetService @Inject()(organizationDAO: OrganizationDAO,
     }
 
   def publicWrites(dataSet: DataSet,
-                   userOpt: Option[User],
+                   requestingUserOpt: Option[User],
                    skipResolutions: Boolean = false,
-                   requestingUserTeamManagerMemberships: Option[List[TeamMembership]] = None): Fox[JsObject] = {
-    implicit val ctx = GlobalAccessContext
+                   requestingUserTeamManagerMemberships: Option[List[TeamMembership]] = None)(
+      implicit ctx: DBAccessContext): Fox[JsObject] =
     for {
-      organization <- organizationDAO.findOne(dataSet._organization) ?~> "organization.notFound"
-      teams <- allowedTeamsFor(dataSet._id)
+      organization <- organizationDAO.findOne(dataSet._organization)(GlobalAccessContext) ?~> "organization.notFound"
+      teams <- allowedTeamsFor(dataSet._id, requestingUserOpt)
       teamsJs <- Fox.serialCombined(teams)(t => teamService.publicWrites(t))
       logoUrl <- logoUrlFor(dataSet, Some(organization))
-      isEditable <- isEditableBy(dataSet, userOpt, requestingUserTeamManagerMemberships)
-      lastUsedByUser <- lastUsedTimeFor(dataSet._id, userOpt)
+      isEditable <- isEditableBy(dataSet, requestingUserOpt, requestingUserTeamManagerMemberships)
+      lastUsedByUser <- lastUsedTimeFor(dataSet._id, requestingUserOpt)
       dataStore <- dataStoreFor(dataSet)
       dataStoreJs <- dataStoreService.publicWrites(dataStore)
       dataSource <- dataSourceFor(dataSet, Some(organization), skipResolutions)
       publicationOpt <- Fox.runOptional(dataSet._publication)(publicationDAO.findOne(_))
-      publicationJson <- Fox.runOptional(publicationOpt)(publicationService.publicWrites(_))
+      publicationJson <- Fox.runOptional(publicationOpt)(publicationService.publicWrites)
     } yield {
       Json.obj(
         "name" -> dataSet.name,
@@ -305,9 +313,9 @@ class DataSetService @Inject()(organizationDAO: OrganizationDAO,
         "sortingKey" -> dataSet.sortingKey,
         "details" -> dataSet.details,
         "publication" -> publicationJson,
+        "isUnreported" -> Json.toJson(dataSource.statusOpt.contains(unreportedStatus)),
         "isForeign" -> dataStore.isForeign
       )
     }
-  }
 
 }
diff --git a/app/utils/WkConf.scala b/app/utils/WkConf.scala
@@ -12,6 +12,7 @@ class WkConf @Inject()(configuration: Configuration) extends ConfigReader {
   object Application {
 
     val insertInitialData = get[Boolean]("application.insertInitialData")
+    val insertLocalConnectDatastore = get[Boolean]("application.insertLocalConnectDatastore")
     val title = get[String]("application.title")
 
     object Authentication {
@@ -125,5 +126,16 @@ class WkConf @Inject()(configuration: Configuration) extends ConfigReader {
   }
 
   val children =
-    List(Application, Http, Mail, WebKnossos, Datastore, User, Braintracing, Features, Silhouette, Airbrake, Google)
+    List(Application,
+         Http,
+         Mail,
+         WebKnossos,
+         Datastore,
+         Tracingstore,
+         User,
+         Braintracing,
+         Features,
+         Silhouette,
+         Airbrake,
+         Google)
 }
diff --git a/conf/application.conf b/conf/application.conf
@@ -64,6 +64,7 @@ braintracing {
 
 application {
   insertInitialData = true
+  insertLocalConnectDatastore = true
   authentication {
     enableDevAutoLogin = false
     enableDevAutoAdmin = false

diff --git a/conf/connect/.gitignore b/conf/connect/.gitignore
@@ -0,0 +1 @@
+datasets.json
diff --git a/conf/connect/config.json b/conf/connect/config.json
@@ -0,0 +1,7 @@
+{
+    "server": {"host": "0.0.0.0", "port": 8000, "url": "http://localhost:8000"},
+    "datastore": {"name": "connect", "key": "secret-key"},
+    "webknossos": {"url": "http://localhost:9000"},
+    "backends": {"neuroglancer": {}},
+    "datasets_path": "data/datasets.json"
+}
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -12,11 +12,14 @@ services:
       - "fossildb-persisted:fossildb"
       - "postgres-persisted:postgres"
       - redis
+      - webknossos-connect
     depends_on:
       postgres-persisted:
         condition: service_healthy
       fossildb-persisted:
         condition: service_healthy
+      webknossos-connect:
+        condition: service_healthy
     command:
       - -Dconfig.file=conf/application.conf
       - -Djava.net.preferIPv4Stack=true
@@ -122,11 +125,14 @@ services:
       - "fossildb-dev:fossildb"
       - "postgres-dev:postgres"
       - redis
+      - webknossos-connect
     depends_on:
       postgres-dev:
         condition: service_healthy
       fossildb-dev:
         condition: service_healthy
+      webknossos-connect:
+        condition: service_healthy
     environment:
       - POSTGRES_URL=jdbc:postgresql://postgres/webknossos
     command:
@@ -166,11 +172,14 @@ services:
       - postgres
       - fossildb
       - redis
+      - webknossos-connect
     depends_on:
       postgres:
         condition: service_healthy
       fossildb:
         condition: service_healthy
+      webknossos-connect:
+        condition: service_healthy
     environment:
       - POSTGRES_URL=jdbc:postgresql://postgres/webknossos_testing
     command:
@@ -195,6 +204,13 @@ services:
       - ".:/home/pptruser/webknossos"
     user: ${USER_UID:-1000}:${USER_GID:-1000}
 
+  # webKnossos-connect
+  webknossos-connect:
+    image: scalableminds/webknossos-connect:master__190
+    volumes:
+      - "./conf/connect:/app/data"
+    network_mode: host
+
   # Postgres
   postgres:
     image: postgres:10-alpine