Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-448 Add data access page #450

Merged
merged 7 commits into from
Jun 29, 2020
Merged

GH-448 Add data access page #450

merged 7 commits into from
Jun 29, 2020

Conversation

sgratiy
Copy link
Contributor

@sgratiy sgratiy commented Jun 23, 2020 •
edited
Loading

Addresses:

issue #448

Type of Fix:

Documentation Change

Tasks:

  • Add data access page
  • Add comment in the gallery of examples to refer to docs (data access page) for downloading data
  • Create utility for accessing file manifest and experiment metadata. This are temporally committed as part of the repo until this data is publicly available through brain-map.org

@sgratiy sgratiy changed the title Gh 448/data access page GH-448 Add data access page Jun 23, 2020
@sgratiy sgratiy changed the base branch from release to dev June 23, 2020 01:01
@sgratiy sgratiy force-pushed the GH-448/data-access-page branch 3 times, most recently from aba9cad to 8999e8d Compare June 23, 2020 17:14
@sgratiy sgratiy changed the base branch from dev to release June 23, 2020 18:39
@sgratiy sgratiy requested review from NileGraddis, gnayuy and MatthewAitken and removed request for gnayuy June 23, 2020 19:34
NileGraddis
NileGraddis reviewed Jun 23, 2020
View reviewed changes
docs/download_data.rst Outdated
and <URL> is the url path of a file or archive.

The paths to individual data files withing the archive are listed in the file manifest
(need links) under the ``arhive_uri`` column.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"archive_uri" typo

Copy link
Contributor

@NileGraddis NileGraddis Jun 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also didn't we get these file manifests last week? Do we need to post them somewhere and insert links? Could they just be committed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

per discussion with @wbwakeman, the manifests will be posted on a "Explore" page on brain-map.org. When this page is live we should be able to link to it.

@NileGraddis
Copy link
Contributor

NileGraddis commented Jun 23, 2020 •
edited
Loading

It seems like we really need to get our hands on these manifest files in order to provide end-to-end download instructions.

Once we do, it might be nice to include a short Pandas/Requests/dandi based snippet that downloads a file based on some criteria and makes an EphysDataSet from it.

@sgratiy
Copy link
Contributor Author

sgratiy commented Jun 24, 2020 •
edited
Loading

Once we do, it might be nice to include a short Pandas/Requests/dandi based snippet that downloads a file based on some criteria and makes an EphysDataSet from it.

I agree that this would be handy. I would suggest though to ask scientists for the use cases before implementing it.

@sgratiy sgratiy mentioned this pull request Jun 25, 2020
Merged
12 tasks
@sgratiy
GH-448 Add data access page to docs
2d60235 
Include instructions for downloading data from DANDI archive
Add page to menu
@sgratiy
GH-448 Direct to docs for downloading data
3927ea7 
Add comment in the gallery examples directing users to docs
@sgratiy
GH-448 Add data release file information
71cd24c 
Add file manifests for both mouse and human data
Add metadata files for both mouse and human data
@sgratiy
GH-448 Add data access utility
891f697 
Add utility function that provides file manifest, experiment metadata
and archive url
@sgratiy sgratiy force-pushed the GH-448/data-access-page branch 2 times, most recently from 3c6bac3 to 879a281 Compare June 26, 2020 23:56
@sgratiy
GH-448 Update data access doc with the utility
879a281 
Add docs describing how to use the utility that provides file manifest and
experiment metadata
@sgratiy
Copy link
Contributor Author

sgratiy commented Jun 27, 2020

@NileGraddis I have added the utility and committed the manifest/metdata. please take a look again, try it out, and merge if acceptable.

@wbwakeman
Copy link

Updated (but still not yet final due to #428 ) versions of the metadata and manifest files are available at https://alleninstitute-my.sharepoint.com/:f:/g/personal/waynew_alleninstitute_org/El0YAZ94AYNHpvODWrTwFloBdV13h-kH9DSKYOMUvEg_CA?e=eoHFAv . Those should be used instead.

Typo: chnage

Grammar: "satisfying a particular experimental conditions." should be "satisfying particular experimental conditions."

@NileGraddis NileGraddis self-assigned this Jun 29, 2020
@NileGraddis
Copy link
Contributor

OK, I've

  1. made the updates/typo fixes that @wbwakeman suggested.
  2. swapped "organism" -> "dataset" in the ARCHIVE_INFO (in case we release future datasets that distinct from these but use the same organisms).
  3. added csvs to the MANIFEST.in so that users who pip install IPFX can use them.

@NileGraddis NileGraddis merged commit ac92c1a into release Jun 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NileGraddis NileGraddis NileGraddis left review comments

@gnayuy gnayuy gnayuy approved these changes

@MatthewAitken MatthewAitken MatthewAitken approved these changes

Assignees

@NileGraddis NileGraddis

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@sgratiy @NileGraddis @wbwakeman @gnayuy @MatthewAitken