Download Sample Datasets #3725

fm3 · 2019-02-04T16:03:58Z

URL of deployed dev instance (used for testing):

https://demodataset.webknossos.xyz

Steps to test:

new routes GET http://localhost:9000/data/datasets/sample/Connectomics_Department?token=secretScmBoyToken and POST http://localhost:9000/data/datasets/sample/Connectomics_Department/Sample_Cremi_dsA_plus/download?token=secretScmBoyToken
corresponding view can be found next to upload dataset (“or add sample dataset”)

Issues:

fixes Pre-package WK with demo datasets #3642

Updated changelog
~~[ ] Updated migration guide if applicable~~
Updated documentation if applicable
Needs datastore update after deployment
Ready for review

…o-dataset

…odal

daniel-wer · 2019-02-14T16:24:16Z

@fm3 I added a first version of a sample datasets modal that shows the available sample datasets and allows to add them to webKnossos. To make it easy to test, I added a button in the dataset view.

I noticed that it seems like the downloading status is never reported. When triggering the download, the status remains "available" for some seconds and then switches to "present" directly. Or maybe the downloading state is only there very briefly (I'm polling every 500ms), ideally it would be set directly after triggering the download :)

The remaining functionality works very well! 🎉

Threre's still stuff missing in the frontend:

Decide where exactly the option to add sample datasets should be available
Choose the correct datastore (or datastores?) to download from (hardcoded as localhost:9000 for now).

fm3 · 2019-02-18T11:04:35Z

Thanks @daniel-wer !
The modal looks good :)
I found the issue causing the “downloading” state to not be reported. That should be fixed now.
Another thing we should consider: if the download fails in the back-end, the status goes back to available (not sure if setting it to failed would really work well, raising the question when that should be resetted). Could the frontend interpret repeated “available”s after the download request was started as “failed”, until the page is reloaded? I’m not sure what behavior I’d prefer here, maybe we can talk about that.

Also, if the first request fails, the download wasn’t started (there are a couple of checks first), which is also not reflected in the modal’s behavior right now, it will be polling indefinitely.

…o-dataset

daniel-wer · 2019-02-18T15:52:06Z

Another thing we should consider: if the download fails in the back-end, the status goes back to available (not sure if setting it to failed would really work well, raising the question when that should be resetted). Could the frontend interpret repeated “available”s after the download request was started as “failed”, until the page is reloaded? I’m not sure what behavior I’d prefer here, maybe we can talk about that.

Is it guaranteed, that when requesting the sample dataset status after triggering the download of dataset X (and having waited for the response), that the status for X === "available" if and only if the download request failed or could it be that the status for X is STILL available, because the internal data structures were not updated yet?
If the former is guaranteed, then this should be pretty easy, otherwise it'll probably get a bit dirty.

Also, if the first request fails, the download wasn’t started (there are a couple of checks first), which is also not reflected in the modal’s behavior right now, it will be polling indefinitely.

I fixed that :)

fm3 · 2019-02-18T16:10:40Z

Is it guaranteed, that when requesting the sample dataset status after triggering the download of dataset X (and having waited for the response), that the status for X === "available" if and only if the download request failed

I shoud think so, yes.

I fixed that :)

Cool, thanks!

fm3 · 2019-02-18T16:32:07Z

~~note to self, todo: add deterministic sorting to dataStoreDAO.findAll~~ done

… to add dataset view which allows to add sample datasets

daniel-wer · 2019-02-20T14:51:08Z

I added the option to add Sample Datasets to a couple of spots (as we discussed):

A link at the bottom of the Add Dataset view, which allows to add sample datasets even when there are existing datasets already.
As a placeholder for the dataset list if there are no datasets, see below:

As part of the onboarding flow - Add Dataset step. I've reworked this step quite a bit to incorporate the sample dataset option (which is the most prominent option by design as it is the easiest), see below:

I think this is ready for a first review round. Any feedback regarding the design, copy and functionality appreciated :)

daniel-wer · 2019-02-20T14:52:48Z

Note that the current backend version expects http://localhost/cremi.zip to be a zipped dataset for testing.
We'll have to think about where we want to host the sample datasets (and which).

fm3 · 2019-02-20T14:55:04Z

the docs link to https://static.webknossos.org/data/e2006_wkw.zip. I think, the cremi one actually also makes sense (small, includes segmentation)

philippotto

Nice work! I'll approve once the proper hosting is done :)

conf/messages

frontend/javascripts/dashboard/dataset/sample_datasets_modal.js

philippotto · 2019-02-21T09:56:35Z

frontend/javascripts/dashboard/dataset/sample_datasets_modal.js

+
+const SampleDatasetsModal = ({ destroy, onOk, organizationName }: Props) => {
+  const [pendingDatasets, setPendingDatasets] = useState([]);
+  const [datastores] = useDatastores();


Could useDatastores directly return datastores without the array wrapper or do hooks dictate/encourage the current way?

Yes it absolutely could (and probably should). At first I wanted to maintain the hooks pattern of returning [value, valueSetter], but that didn't really work out ^^

philippotto · 2019-02-21T10:09:24Z

frontend/javascripts/dashboard/dataset/sample_datasets_modal.js

+
+  useInterval(fetchDatasets, pendingDatasets.length ? 1000 : null);
+
+  const handleSampleDatasetDownload = async (name: string) => {


I'm wondering whether it makes sense to move handleSampleDatasetDownload, the pendingDatasets definition and the useDatastores call into the useSampleDatasets hook. At the moment, it feels a bit weird that failedDatasets are provided by the useSampleDatasets hook, but the pendingDatasets have to be maintained at an higher level.
handleSampleDatasetDownload would be another return argument of useSampleDatasets then.

This would clean up the view logic here a bit, but obviously the useSampleDatasets hook would get longer. I'd argue, that the abstraction level is more "even", though. What do you think?

I struggled with that as well, thinking along the same lines ^^ I'll give this option a try and see what I can do to clean this up a little bit :)

philippotto · 2019-02-21T10:16:53Z

frontend/javascripts/dashboard/dataset/sample_datasets_modal.js

+  organizationName: string,
+};
+
+function useDatastores(): [Array<APIDataStore>] {


I imagine that these ten lines of code will be useful quite a lot in the future (independent of entity type, of course). We don't have to do it now, but I'd think that something like const datastores = useAsyncGet(getDataStores, []) would be a good abstraction (naming is debatable).

I created a generic useFetch method and included a third parameter for the dependencies (for useEffect) :)

…o-dataset

normanrz · 2019-02-26T13:43:48Z

https://static.webknossos.org/data/e2006_wkw.zip
Raw SBEM data and segmentation (sample cutout)
Connectomic reconstruction of the inner plexiform layer in the mouse retina
M Helmstaedter, KL Briggman, S Turaga, V Jain, HS Seung, W Denk.
Nature. 08 August 2013. https://doi.org/10.1038/nature12346
https://static.webknossos.org/data/FD0144_wkw.zip
Raw SBEM data and segmentation (sample cutout)
FluoEM, virtual labeling of axons in three-dimensional electron microscopy data for long-range connectomics
F Drawitsch, A Karimi, KM Boergens, M Helmstaedter.
eLife. 14 August 2018. https://doi.org/10.7554/eLife.38976
https://static.webknossos.org/data/MPRAGE_250um.zip
MRI data
T1-weighted in vivo human whole brain MRI dataset with an ultrahigh isotropic resolution of 250 μm
F Lüsebrink, A Sciarra, H Mattern, R Yakupov, O Speck
Scientific Data. 14 March 2017. https://doi.org/10.1038/sdata.2017.32

…loads still running works as expected

…o-dataset

fm3 · 2019-02-27T12:15:47Z

@daniel-wer I added the three datasets from above. Could you display the description in the front-end? I added it as description in the json. It contains line-breaks, not sure how that is best handled.

…o-dataset

daniel-wer · 2019-02-28T11:24:25Z

@fm3 I took care of the descriptions :)
During testing of the sample datasets, I noticed that the Sample_FD0144_wkw dataset cannot be openend, because no resolutions are sent as part of the dataset request. The datasource config (in the Edit view) does contain resolutions, though, so I'm not sure what the problem is. Maybe you could have a look at that?

fm3 · 2019-02-28T12:43:20Z

There’s two layers named “color_2” in the datasource-properties.json in FD0144_wkw.zip. @normanrz could you update it? I suppose one of them should be “color_3”. Error handling for this is not great, but I don’t know a quick way to fix that. Created #3845 to track.

backend: 2019-02-28 13:33:59,110 [ERROR] models.binary.DataSetDataLayerDAO - SQL Error: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "dataset_layers_pkey"
backend:   Detail: Key (_dataset, name)=(5c77d537640100b601f3b0b5, color_2) already exists.

fm3 · 2019-03-04T12:46:07Z

Thanks for fixing that @normanrz! one last thing: the real bounding box for that dataset (FD0144_wkw) seems to be smaller (I don’t see data for z>=164) so to optimize the user experience we might want to add that to the datasource-properties.json. Shouldn’t block the PR, though

normanrz · 2019-03-04T16:14:43Z

fixed

fm3 · 2019-03-05T07:42:37Z

Cool, thanks! I updated the docs. If you have no objections @philippotto @normanrz I suggest we can merge this today

philippotto · 2019-03-06T10:09:53Z

docs/getting_started.md

-Identify a dataset that your interested in and click on `Start Skeleton Tracing` to create a new skeleton annotation. 
-webKnossos will launch the main annotation screen allowing you to navigate your dataset and place markers to reconstruct skeletons. 
+To get started with your first annotation, navigate to the `Datasets` tab on your [dashboard](./dashboard.md).
+Identify a dataset that your interested in and click on `Start Skeleton Tracing` to create a new skeleton annotation.


I know that you only changed the whitespace here, but it should be you are 🙈

philippotto · 2019-03-06T10:10:50Z

Docs look good to me 👍

[WIP] download demo dataset

567ab8f

fm3 self-assigned this Feb 4, 2019

fm3 added 3 commits February 11, 2019 17:15

prepare download + guard

ab27fcd

Merge branch 'master' into demo-dataset

0c95bcd

list available sample datasets, enable download

0df51af

fm3 changed the title ~~[WIP] Download Demo dataset~~ [WIP] Download Sample datasets Feb 12, 2019

fm3 assigned daniel-wer Feb 12, 2019

daniel-wer added 3 commits February 14, 2019 13:17

Merge branch 'master' of github.com:scalableminds/webknossos into dem…

d7a243e

…o-dataset

add sample datasets modal, add temp button to datasets view to open m…

e2ee248

…odal

fix typo in downloading status string

7c1e3ac

fix reporting of downloading state

16fdaa5

daniel-wer added 2 commits February 18, 2019 14:52

Merge branch 'master' of github.com:scalableminds/webknossos into dem…

8fba42d

…o-dataset

use all datastores for sample dataset downloads, fix error case

d2a57be

fm3 and others added 3 commits February 19, 2019 11:01

deterministic sorting for datastore list (by name)

7f4884b

use single datastore, detect failed datasets, refactoring

1be1b42

rework onboarding dataset step and dataset list placeholder, add link…

06609aa

… to add dataset view which allows to add sample datasets

daniel-wer requested a review from philippotto February 20, 2019 14:50

philippotto reviewed Feb 21, 2019

View reviewed changes

fm3 and others added 4 commits February 21, 2019 11:41

Update messages

a268dc9

merge master into demo-dataset

756d4fb

Merge branch 'master' of github.com:scalableminds/webknossos into dem…

c49628d

…o-dataset

refactor sample dataset modal state management according to PR feedback

9bf5e61

daniel-wer and others added 4 commits February 26, 2019 15:04

create generic useFetch method, make sure opening the modal with down…

054e84b

…loads still running works as expected

Merge branch 'master' of github.com:scalableminds/webknossos into dem…

155825a

…o-dataset

small style tweaks for onboarding dataset view

23caa4c

insert actual sample datasets

75caaea

daniel-wer added 3 commits February 28, 2019 11:21

add description to sample datasets modal - untested

93ac035

preserve whitespace in sample datasets modal

b0d5b50

Merge branch 'master' of github.com:scalableminds/webknossos into dem…

2c7564e

…o-dataset

fm3 changed the title ~~[WIP] Download Sample datasets~~ Download Sample Datasets Mar 4, 2019

philippotto approved these changes Mar 4, 2019

View reviewed changes

Merge branch 'master' into demo-dataset

c9cf39f

fm3 added 2 commits March 5, 2019 08:38

update docs + changelog

171b681

merge

835748d

merge master into demo-dataset

96d046b

fm3 added backend frontend labels Mar 5, 2019

Merge branch 'master' into demo-dataset

5300199

philippotto reviewed Mar 6, 2019

View reviewed changes

fm3 merged commit 5e7016f into master Mar 6, 2019

fm3 deleted the demo-dataset branch March 6, 2019 10:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download Sample Datasets #3725

Download Sample Datasets #3725

fm3 commented Feb 4, 2019 •

edited by daniel-wer

Loading

daniel-wer commented Feb 14, 2019 •

edited

Loading

fm3 commented Feb 18, 2019

daniel-wer commented Feb 18, 2019

fm3 commented Feb 18, 2019

fm3 commented Feb 18, 2019 •

edited

Loading

daniel-wer commented Feb 20, 2019

daniel-wer commented Feb 20, 2019

fm3 commented Feb 20, 2019

philippotto left a comment

philippotto Feb 21, 2019

daniel-wer Feb 25, 2019

philippotto Feb 21, 2019 •

edited

Loading

daniel-wer Feb 25, 2019

philippotto Feb 21, 2019

daniel-wer Feb 26, 2019

normanrz commented Feb 26, 2019

fm3 commented Feb 27, 2019

daniel-wer commented Feb 28, 2019

fm3 commented Feb 28, 2019

fm3 commented Mar 4, 2019

normanrz commented Mar 4, 2019

fm3 commented Mar 5, 2019

philippotto Mar 6, 2019

philippotto commented Mar 6, 2019


		useInterval(fetchDatasets, pendingDatasets.length ? 1000 : null);

		const handleSampleDatasetDownload = async (name: string) => {

Download Sample Datasets #3725

Download Sample Datasets #3725

Conversation

fm3 commented Feb 4, 2019 • edited by daniel-wer Loading

URL of deployed dev instance (used for testing):

Steps to test:

Issues:

daniel-wer commented Feb 14, 2019 • edited Loading

fm3 commented Feb 18, 2019

daniel-wer commented Feb 18, 2019

fm3 commented Feb 18, 2019

fm3 commented Feb 18, 2019 • edited Loading

daniel-wer commented Feb 20, 2019

daniel-wer commented Feb 20, 2019

fm3 commented Feb 20, 2019

philippotto left a comment

Choose a reason for hiding this comment

philippotto Feb 21, 2019

Choose a reason for hiding this comment

daniel-wer Feb 25, 2019

Choose a reason for hiding this comment

philippotto Feb 21, 2019 • edited Loading

Choose a reason for hiding this comment

daniel-wer Feb 25, 2019

Choose a reason for hiding this comment

philippotto Feb 21, 2019

Choose a reason for hiding this comment

daniel-wer Feb 26, 2019

Choose a reason for hiding this comment

normanrz commented Feb 26, 2019

fm3 commented Feb 27, 2019

daniel-wer commented Feb 28, 2019

fm3 commented Feb 28, 2019

fm3 commented Mar 4, 2019

normanrz commented Mar 4, 2019

fm3 commented Mar 5, 2019

philippotto Mar 6, 2019

Choose a reason for hiding this comment

philippotto commented Mar 6, 2019

fm3 commented Feb 4, 2019 •

edited by daniel-wer

Loading

daniel-wer commented Feb 14, 2019 •

edited

Loading

fm3 commented Feb 18, 2019 •

edited

Loading

philippotto Feb 21, 2019 •

edited

Loading