Folder And Dataset Metadata #7886

MichaelBuessemeyer · 2024-06-17T15:48:51Z

URL of deployed dev instance (used for testing):

https://folderdatasetproperties.webknossos.xyz/

Steps to test:

Open the dashboard & create another folder.
Try adding metadata to the folder. Try out all three kinds of entries. Test whether autosave works
Try editing metadata for a dataset
Switch between datasets, the metadata should stay consistent
Switch between folders, the metadata should stay consistent
And also switch between folders and datasets and also look for any errors / inconsistencies / lost metadata entries.

TODOs:

Clean up git history
Allow to search for metadata props -> Other issue Allow filtering datasets by Metadata #8034
Add props & UI and everything for folders
Update UI to the newest design proposal
Clean up code
Test dataset update route without supplying the metadata field
- Works. Tested with firefox's edit and resend functionality.

Open Question

What about supporting dataset tags? Should they be removed? Should one still be able to update the dataset tags via the API? Should they be removed from the schema? Should they be transferred into metadata entries?
How should tags be transferred into metadata?
- As a single string array entry with key "tags"
- For each on entry with numbered keys like "tagX" and x is the index of the tag in the tag array
- other ideas

Issues:

fixes #5968

(Please delete unneeded items, merge only when none are left open)

Updated changelog
Updated migration guide if applicable
Updated documentation if applicable
Adapted wk-libs python client if relevant API parts change
Removed dev-only changes like prints and application.conf edits
Considered common edge cases
Needs datastore update after deployment

* make login, registration and forgot password view more responsive to mobile devices - includes adding a link back to login for password reset view * make dashboard dataset view less awful * add changelog entry

* allow viewing a mapping * WIP: enable setting default active mappings in default view configuration * add default active mapping to default view configuration - add config settings to dataset settings default view config tab - auto load initial default mapping * fix bottom margin of layer settings - was broken in view mode with multiple segmentation layers; the "Save View Config..." Button had no spacing to towards * fix having the mapping settings enabled for layers with default mapping * WIP: move default mapping store proeprty to dataset layer config - undo moving mapping selection to a shared component * moved default active mapping from separate select to view config layer config - Also added validation check whether the mapping exists in the dataset view config settings * allow saving no default active mapping from view mode & model init to new dataset config version * rename defaultMapping to mapping * cache available mappings fetched from backend when checking config in dataset settings * remove logging statements * clean up * add changelog entry * apply pr feedback

* fix animation modal color layer validation * changelog * rename variable * apply pr feedback * improve error message of uint24 layers in animation modal --------- Co-authored-by: Michael Büßemeyer <[email protected]>

* validate BB size to be larger then 0 * changelog * appy PR feedback

…lder-dataset-properties

…aset

…ontend dataset filtering - reason: ll.530 checks for sortedInfo.columnKey == null and columnKey is per default "" which makes this check fail and thus the chained map will never be executed

…der-dataset-properties

…e metadata table

…der-dataset-properties

… WIP restyling of metadata / details table

…ent types

fm3

Backend looking good already :) I added a couple of comments.
I also tested a bit more and it’s looking pretty good :)

I did not yet review the migrations. I think we need to decide about the NULLability first, and also about what to do with the tags.

If I understand correctly, the dataset tags are now turned into the new metadata. If this is true, the tags should be removed from the SQL schema and the Dataset case class.

However, the dataset json still needs to serve the tags for the moment, until the libs client is adapted. I think it would be good if the old tags could be served there. Meaning that the json serialization should convert the relevant metadata to tags and serve them, alongside the new metadata attribute. In the same way, updating the tags from the libs should also stay possible, so an adapter for that is needed as well.

Also, could you create a libs issue to adapt the libs to the new metadata format and deprecate the tags there too?

Also note that #8006 was merged first so that the schema version in this PR needs to be increased to avoid conflicts.

fm3 · 2024-08-20T11:46:55Z

app/controllers/DatasetController.scala

          for {
            dataset <- datasetDAO.findOneByNameAndOrganization(datasetName, request.identity._organization) ?~> notFoundMessage(
              datasetName) ~> NOT_FOUND
+            maybeUpdatedMetadata = if (metadata.isDefined) metadata else dataset.metadata


should use getOrElse or just orElse if an option is required. I’d also inline it below, just like folderId.getOrElse(dataset._folder)

app/models/dataset/Dataset.scala

fm3 · 2024-08-20T11:49:23Z

app/models/folder/Folder.scala

-    Fox.successful(Folder(ObjectId(r._Id), r.name))
+    for {
+      metadata <- parseMetadata(r.metadata)
+      folder <- Fox.successful(Folder(ObjectId(r._Id), r.name, metadata))


Suggested change

folder <- Fox.successful(Folder(ObjectId(r._Id), r.name, metadata))

folder = Folder(ObjectId(r._Id), r.name, metadata)

fm3 · 2024-08-20T11:49:49Z

app/models/folder/Folder.scala

+  private def parseWithParent(t: (String, String, Option[String], Option[String])): Fox[FolderWithParent] =
+    for {
+      metadata <- parseMetadata(t._3)
+      folderWithParent <- Fox.successful(FolderWithParent(ObjectId(t._1), t._2, metadata, t._4.map(ObjectId(_))))


Suggested change

folderWithParent <- Fox.successful(FolderWithParent(ObjectId(t._1), t._2, metadata, t._4.map(ObjectId(_))))

folderWithParent = FolderWithParent(ObjectId(t._1), t._2, metadata, t._4.map(ObjectId(_)))

fm3 · 2024-08-20T11:52:45Z

tools/postgres/schema.sql

@@ -123,14 +123,14 @@ CREATE TABLE webknossos.datasets(
  sharingToken CHAR(256),
  logoUrl VARCHAR(2048),
  sortingKey TIMESTAMPTZ NOT NULL,
-  details JSONB,
+  metadata JSONB DEFAULT '[]',


Why the DEFAULT '[]'? The backend code seems to be able to handle NULL/None. If we want to go with DEFAULT '[]', we should do so everywhere and make the property NOT NULL in sql and also non-optional in scala/ts code

🤔 My thought here was that it might be easier to work with at least an empty array instead of null. I added the null / None support to the code to ensure easier compatibility. This includes update calls not necessarily having to pass metadata (at least I hope that's the case, as I wanted to implement it that way).

But I can easily remove the default value from the column :) Or even go further and make metadata non-optional if you think this is preferable.

I don’t really have a preference for default value versus nullable. But I don’t much like having both. Please pick what’s easier implementation-wise :) My guess is that removing the default value (and so have the default be NULL/None) is easier.

Not is should be non nullable and with DEFAULT '[] for dataset and folders :).

The update route for datasets parses the metadata field as optional and used the already existing metadata in the backend query in case the user / client did not provide any metadata (untested) 🙈

…der-dataset-properties

MichaelBuessemeyer · 2024-08-21T12:37:45Z

Ok what to do with tags :)

I just talked with @normanrz and the conclusion is to keep tags as some are using them as well as the pretty important filtering feature.

If we later want to merge the tags into the metadata, we can still do that. In case dataset/folder already has a metadata field named "tags" already exists, we just come up with something :).

MichaelBuessemeyer · 2024-08-22T05:24:00Z

frontend/javascripts/dashboard/folders/details_sidebar.tsx

@@ -172,15 +177,15 @@ function DatasetDetails({ selectedDataset }: { selectedDataset: APIDatasetCompac
            </Tag>
          )}
        </div>
+        {fullDataset && (


Open question? Is it ok to move the tags into the "spinner area"? Previously they weren'T but, the metadata should be within the "spinner area" as it depends on fullDataset and IMO the newly added UI element should appear after the already known tags.

sure, why not 👍

MichaelBuessemeyer · 2024-08-22T08:22:34Z

testing went well. Tags should be restored in the frontend :)

…der-dataset-properties

fm3

Backend LGTM

philippotto

just tested the dev instance again and everything looks good 👍 I didn't test the migration. ifi that is tested well, feel free to merge and deploy

MichaelBuessemeyer

@fm3 I just pushed a commit to fix the reversion. Could you please give me a final thumbs up?

I just tested the migration and reversion. both should work

MichaelBuessemeyer · 2024-08-27T11:07:18Z

conf/evolutions/reversions/119-add-metadata-to-folders-and-datasets.sql

+
+-- Add existing info on species of metadata to details
+UPDATE webknossos.datasets
+SET details = jsonb_set(COALESCE(details, '{}'), '{species}', (


The newest commit changed

SET details = jsonb_set(details, '{species}', (

to

SET details = jsonb_set(COALESCE(details, '{}'), '{species}', (

This was necessary, as the default value of the newly re-added details column was null and thus the jsonb_set would also return null. This should work now :)

MichaelBuessemeyer · 2024-08-27T13:23:37Z

And @fm3 no need to test the migration again. I tested it and it should work ™️ 🙈

fm3

🎉
(Could you please link the follow-up issue for search in the PR description?)

MichaelBuessemeyer added backend frontend labels Jun 17, 2024

MichaelBuessemeyer self-assigned this Jun 17, 2024

MichaelBuessemeyer and others added 6 commits June 17, 2024 18:33

Add dataset metadata input to details view in dashboard

dbe80eb

update docs

4ab38dc

Improve mobile support of views (#7876)

242cf1d

* make login, registration and forgot password view more responsive to mobile devices - includes adding a link back to login for password reset view * make dashboard dataset view less awful * add changelog entry

Fix animation modal color layer validation (#7882)

5bb3997

* fix animation modal color layer validation * changelog * rename variable * apply pr feedback * improve error message of uint24 layers in animation modal --------- Co-authored-by: Michael Büßemeyer <[email protected]>

Validate animation job bounding box (#7883)

1797a4e

* validate BB size to be larger then 0 * changelog * appy PR feedback

MichaelBuessemeyer force-pushed the folder-dataset-properties branch from 73ae88a to 1797a4e Compare June 17, 2024 16:33

fm3 changed the title ~~Folder dataset properties~~ Folder And Dataset Properties Jun 18, 2024

MichaelBuessemeyer added 3 commits June 18, 2024 10:25

Merge branch 'master' of github.com:scalableminds/webknossos into fo…

55c3d5f

…lder-dataset-properties

force serialization of dataset details prop in compact version of dat…

5cf5bfd

…aset

WIP adding details to folder

70b185e

MichaelBuessemeyer added the new feature label Jun 19, 2024

MichaelBuessemeyer added 15 commits June 19, 2024 13:02

First version metadata support for folders

3c4ae12

fix folder listing query

434b875

finish frontend support for folder metadata

47daad4

default to undefined columnKey and not null to potentially prevent fr…

9a8a5cd

…ontend dataset filtering - reason: ll.530 checks for sortedInfo.columnKey == null and columnKey is per default "" which makes this check fail and thus the chained map will never be executed

add evolution

0bd1c29

update test db init data & folder snapshot test & bump schema version

51cd0e4

Merge branch 'master' of github.com:scalableminds/webknossos into fol…

488f5bd

…der-dataset-properties

remove unused import

b3294c4

also blur on property input

8051729

only support / show current error

3fb9613

improve css

1f0db40

WIP: use new json schema to save dataset & folder details and re-styl…

eaaaaea

…e metadata table

Merge branch 'master' of github.com:scalableminds/webknossos into fol…

4ef393a

…der-dataset-properties

fix backend to work with new type of details dataset / folder field &…

846b500

… WIP restyling of metadata / details table

WIP: improve dataset details table styling and add support for differ…

e7ed7f8

…ent types

fm3 requested changes Aug 20, 2024

View reviewed changes

MichaelBuessemeyer added 3 commits August 21, 2024 14:13

Merge branch 'master' of github.com:scalableminds/webknossos into fol…

e9fccae

…der-dataset-properties

update schema migration version

984b889

apply feedback

a5184ee

MichaelBuessemeyer commented Aug 22, 2024

View reviewed changes

MichaelBuessemeyer added 2 commits August 22, 2024 07:28

readd tag support in frontend (and backend)

3ea1f4d

fix backend

30bb3e9

markbader mentioned this pull request Aug 22, 2024

Enable metadata access for remote datasets. scalableminds/webknossos-libs#1163

Merged

3 tasks

MichaelBuessemeyer added 2 commits August 22, 2024 16:15

make metadata non nullable

6080226

update snapshots tests

bc6739d

MichaelBuessemeyer requested a review from fm3 August 23, 2024 11:10

MichaelBuessemeyer added 2 commits August 23, 2024 13:33

Merge branch 'master' of github.com:scalableminds/webknossos into fol…

32f84d6

…der-dataset-properties

fix metadatatype type definition

6508c59

fm3 reviewed Aug 26, 2024

View reviewed changes

Merge branch 'master' into folder-dataset-properties

de0768a

philippotto approved these changes Aug 26, 2024

View reviewed changes

philippotto and others added 2 commits August 26, 2024 14:10

Merge branch 'master' into folder-dataset-properties

6fffde2

fix reversion against default null values of re-added details column

46224ba

MichaelBuessemeyer commented Aug 27, 2024

View reviewed changes

Merge branch 'master' into folder-dataset-properties

c505be0

fm3 approved these changes Aug 28, 2024

View reviewed changes

philippotto and others added 2 commits August 28, 2024 09:51

Merge branch 'master' into folder-dataset-properties

5dd4872

Merge branch 'master' into folder-dataset-properties

57311df

MichaelBuessemeyer mentioned this pull request Aug 28, 2024

Allow filtering datasets by Metadata #8034

Open

8 tasks

Merge branch 'master' into folder-dataset-properties

76c2a97

MichaelBuessemeyer merged commit 36e3262 into master Aug 28, 2024
2 checks passed

MichaelBuessemeyer deleted the folder-dataset-properties branch August 28, 2024 11:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Folder And Dataset Metadata #7886

Folder And Dataset Metadata #7886

MichaelBuessemeyer commented Jun 17, 2024 •

edited

Loading

fm3 left a comment

fm3 Aug 20, 2024

fm3 Aug 20, 2024

fm3 Aug 20, 2024

fm3 Aug 20, 2024

MichaelBuessemeyer Aug 21, 2024

fm3 Aug 22, 2024

MichaelBuessemeyer Aug 22, 2024

MichaelBuessemeyer commented Aug 21, 2024

MichaelBuessemeyer Aug 22, 2024

philippotto Aug 22, 2024

MichaelBuessemeyer commented Aug 22, 2024

fm3 left a comment

philippotto left a comment

MichaelBuessemeyer left a comment

MichaelBuessemeyer Aug 27, 2024

MichaelBuessemeyer commented Aug 27, 2024

fm3 left a comment

	folder <- Fox.successful(Folder(ObjectId(r._Id), r.name, metadata))
	folder = Folder(ObjectId(r._Id), r.name, metadata)

	folderWithParent <- Fox.successful(FolderWithParent(ObjectId(t._1), t._2, metadata, t._4.map(ObjectId(_))))
	folderWithParent = FolderWithParent(ObjectId(t._1), t._2, metadata, t._4.map(ObjectId(_)))

Folder And Dataset Metadata #7886

Folder And Dataset Metadata #7886

Conversation

MichaelBuessemeyer commented Jun 17, 2024 • edited Loading

URL of deployed dev instance (used for testing):

Steps to test:

TODOs:

Open Question

Issues:

fm3 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelBuessemeyer commented Aug 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelBuessemeyer commented Aug 22, 2024

fm3 left a comment

Choose a reason for hiding this comment

philippotto left a comment

Choose a reason for hiding this comment

MichaelBuessemeyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelBuessemeyer commented Aug 27, 2024

fm3 left a comment

Choose a reason for hiding this comment

MichaelBuessemeyer commented Jun 17, 2024 •

edited

Loading