Look up annotation dataset by tracing id instead of (broken) organizationName field #6548

fm3 · 2022-10-12T08:49:50Z

The organizationName field in the Tracing proto objects is used to look up the dataset.
Unfortunately, in createExplorational, not the dataset organization but the user organization was entered into that field, breaking the features which rely on this lookup for all annotations where the dataset orga is not the same as the user orga.
Those features include:

Ad-Hoc meshing with fallback layer
Zarr streming
Editable Mappings
Redownsampling

Fixing this in createExplorational will not repair existing tracing objects, mutating them in the fossildb is tricky. We should discuss how to proceed here:

(A) Do nothing, these features will stay unavailable for foreign-datset annotations created before the fix
(B) In the dataset lookup, on failure, look for datasets of the same name of other organizations that the user may read, and use those. This should repair most annotations, except where datasets of the same name exist in multiple organizations
(C) Create a python script that fixes this in the fossildb retroactively for all tracing versions (hard, risky)
(D) Create adapter code in wk that repairs all annotations when opening them (creates update actions)
Edit: (E) from comment below: ditch the organizationName and datasetName fields entirely and perform the (cached) lookup each time by the tracing id (the wk-side postgres table should be able to connect tracing id and dataset via foreign keys)

This PR implements E

TODO

caching
cleanup
double-check that front-end does not use the relevant fields in proto
nml writer: pass in dataset name

URL of deployed dev instance (used for testing):

https://___.webknossos.xyz

Steps to test:

Test the four features described above with a multi-organization setup on a public dataset of orga A, with a user of orga B.

Issues:

Updated (unreleased) changelog
Needs datastore update after deployment
Ready for review

fm3 · 2022-10-12T09:05:40Z

...tastore/app/com/scalableminds/webknossos/datastore/services/DSRemoteTracingstoreClient.scala

@@ -40,7 +40,7 @@ class DSRemoteTracingstoreClient @Inject()(
                     cxyz: String,
                     tracingStoreUri: String,
                     token: Option[String]): Fox[Array[Byte]] =
-    rpc(s"$tracingStoreUri/tracings/volume/zarr/$tracingId/$mag/$cxyz")
+    rpc(s"$tracingStoreUri/tracings/volume/zarr/$tracingId/$mag/$cxyz").silent


This slipped in here: avoid logging for every chunk request

fm3 · 2022-10-12T09:54:37Z

Discussion result:
(E) ditch the organizationName and datasetName fields entirely and perform the (cached) lookup each time by the tracing id (the wk-side postgres table should be able to connect tracing id and dataset via foreign keys)

jstriebel

LGTM (besides the two open TODOs) 👍

…/webknossos into foreign-dataset-annotation

…ebar_button * 'master' of github.com:scalableminds/webknossos: added more usage shortcuts to status bar (#6549) Look up annotation dataset by tracing id instead of (broken) organizationName field (#6548) Add links to resource creation pages (#6513) Improve Zarr Import UI (and general editing of datasets) (#6485) Remove unused dataSetName and organizationName property of protobuf objects (#6559) WIP: Added manual task assignment (#6551) Fix zarr private links for volume tracings and some minor improvements to private links UI (#6541) Fix stack overflow in compactMovedNodesAndEdges (#6557)

…ect_spinner * 'master' of github.com:scalableminds/webknossos: added more usage shortcuts to status bar (#6549) Look up annotation dataset by tracing id instead of (broken) organizationName field (#6548) Add links to resource creation pages (#6513) Improve Zarr Import UI (and general editing of datasets) (#6485) Remove unused dataSetName and organizationName property of protobuf objects (#6559) WIP: Added manual task assignment (#6551) Fix zarr private links for volume tracings and some minor improvements to private links UI (#6541) Fix stack overflow in compactMovedNodesAndEdges (#6557)

…ault_volume_layer * 'master' of github.com:scalableminds/webknossos: (21 commits) adds ETA calculation to voxelytics tasks (#6564) update immutability-helper package to v3.1.1 with TS support (#6565) [Skeletons] Allow deletion of root group (#6553) fix disappearing sidebar button (#6555) Enable "What's new" in navbar for everybody (#6563) Highlight Menu in Navbar Menu (#6558) refactor Select loading spinner property (#6556) Reload histogram when reloading layer (#6537) added more usage shortcuts to status bar (#6549) Look up annotation dataset by tracing id instead of (broken) organizationName field (#6548) Add links to resource creation pages (#6513) Improve Zarr Import UI (and general editing of datasets) (#6485) Remove unused dataSetName and organizationName property of protobuf objects (#6559) WIP: Added manual task assignment (#6551) Fix zarr private links for volume tracings and some minor improvements to private links UI (#6541) Fix stack overflow in compactMovedNodesAndEdges (#6557) Add backspace keyboard shortcut to delete active node (#6554) fix scaling in new meshfile rendering (#6552) Added more filters and sorting options to Projects list (#6550) fix double occurance of "ago" in VX progress times (#6535) ...

fm3 added 2 commits October 12, 2022 10:48

Fix organization name field for annotations of foreign-orga datasets

4f27ceb

undo unnecessary change

a9ae435

fm3 added bug backend labels Oct 12, 2022

fm3 self-assigned this Oct 12, 2022

pretty

2d99c32

fm3 added the discussion label Oct 12, 2022

fm3 commented Oct 12, 2022

View reviewed changes

fm3 added 4 commits October 13, 2022 11:50

[WIP] ask wk for datasource by tracing id instead

e1a5e8c

implement routes on wk-side

251b0a4

include annotation temporary store in lookup

5b6e2ef

add comment in proto definition file

0737293

fm3 changed the title ~~Fix organization name field for annotations of foreign-orga datasets~~ Look up annotation dataset by tracing id instead of (broken) organizationName field Oct 13, 2022

fm3 added 3 commits October 13, 2022 14:01

add some caching

1e28ae1

use correct tracing id when downsampling a volume tracing

120724e

changelog

b9a6314

fm3 marked this pull request as ready for review October 13, 2022 13:14

Merge branch 'master' into foreign-dataset-annotation

b9c257a

fm3 requested a review from jstriebel October 13, 2022 13:15

jstriebel approved these changes Oct 13, 2022

View reviewed changes

fm3 and others added 5 commits October 17, 2022 13:56

pass dataset name to nml writer

4e1d061

Merge branch 'master' into foreign-dataset-annotation

b692660

fix parameter order

1967627

Merge branch 'foreign-dataset-annotation' of github.com:scalableminds…

b93b36b

…/webknossos into foreign-dataset-annotation

Merge branch 'master' into foreign-dataset-annotation

6c01381

fm3 merged commit f211d89 into master Oct 18, 2022

fm3 deleted the foreign-dataset-annotation branch October 18, 2022 08:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Look up annotation dataset by tracing id instead of (broken) organizationName field #6548

Look up annotation dataset by tracing id instead of (broken) organizationName field #6548

fm3 commented Oct 12, 2022 •

edited

Loading

fm3 Oct 12, 2022

fm3 commented Oct 12, 2022

jstriebel left a comment

Look up annotation dataset by tracing id instead of (broken) organizationName field #6548

Look up annotation dataset by tracing id instead of (broken) organizationName field #6548

Conversation

fm3 commented Oct 12, 2022 • edited Loading

TODO

URL of deployed dev instance (used for testing):

Steps to test:

Issues:

fm3 Oct 12, 2022

Choose a reason for hiding this comment

fm3 commented Oct 12, 2022

jstriebel left a comment

Choose a reason for hiding this comment

fm3 commented Oct 12, 2022 •

edited

Loading