Skip to content

Commit

Permalink
added NWB conversion tool description to the JOSS manuscript
Browse files Browse the repository at this point in the history
  • Loading branch information
calebweinreb committed Jan 28, 2024
1 parent 1f83ce6 commit 41c1614
Show file tree
Hide file tree
Showing 2 changed files with 105 additions and 81 deletions.
180 changes: 101 additions & 79 deletions paper.bib
Original file line number Diff line number Diff line change
@@ -1,102 +1,124 @@
@software{vispy,
author = {Luke Campagnola and
Eric Larson and
Almar Klein and
David Hoese and
Siddharth and
Cyrille Rossant and
Adam Griffiths and
Nicolas P. Rougier and
asnt and
Kai Mühlbauer and
Alexander Taylor and
MSS and
Talley Lambert and
sylm21 and
Alex J. Champandard and
Max Hunter and
Thomas Robitaille and
Mustafa Furkan Kaptan and
Elliott Sales de Andrade and
Karl Czajkowski and
Lorenzo Gaifas and
Alessandro Bacchini and
Guillaume Favelier and
Etienne Combrisson and
ThenTech and
fschill and
Mark Harfouche and
Michael Aye and
Casper van Elteren and
Cedric GESTES},
title = {vispy/vispy: Version 0.11.0},
month = jul,
year = 2022,
publisher = {Zenodo},
version = {v0.11.0},
doi = {10.5281/zenodo.6795163},
url = {https://doi.org/10.5281/zenodo.6795163}
author = {Luke Campagnola and
Eric Larson and
Almar Klein and
David Hoese and
Siddharth and
Cyrille Rossant and
Adam Griffiths and
Nicolas P. Rougier and
asnt and
Kai Mühlbauer and
Alexander Taylor and
MSS and
Talley Lambert and
sylm21 and
Alex J. Champandard and
Max Hunter and
Thomas Robitaille and
Mustafa Furkan Kaptan and
Elliott Sales de Andrade and
Karl Czajkowski and
Lorenzo Gaifas and
Alessandro Bacchini and
Guillaume Favelier and
Etienne Combrisson and
ThenTech and
fschill and
Mark Harfouche and
Michael Aye and
Casper van Elteren and
Cedric GESTES},
title = {vispy/vispy: Version 0.11.0},
month = jul,
year = 2022,
publisher = {Zenodo},
version = {v0.11.0},
doi = {10.5281/zenodo.6795163},
url = {https://doi.org/10.5281/zenodo.6795163}
}



@article{bento,
abstract = {The study of naturalistic social behavior requires quantification of animals' interactions. This is generally done through manual annotation---a highly time-consuming and tedious process. Recent advances in computer vision enable tracking the pose (posture) of freely behaving animals. However, automatically and accurately classifying complex social behaviors remains technically challenging. We introduce the Mouse Action Recognition System (MARS), an automated pipeline for pose estimation and behavior quantification in pairs of freely interacting mice. We compare MARS's annotations to human annotations and find that MARS's pose estimation and behavior classification achieve human-level performance. We also release the pose and annotation datasets used to train MARS to serve as community benchmarks and resources. Finally, we introduce the Behavior Ensemble and Neural Trajectory Observatory (BENTO), a graphical user interface for analysis of multimodal neuroscience datasets. Together, MARS and BENTO provide an end-to-end pipeline for behavior data extraction and analysis in a package that is user-friendly and easily modifiable.},
article_type = {journal},
author = {Segalin, Cristina and Williams, Jalani and Karigo, Tomomi and Hui, May and Zelikowsky, Moriel and Sun, Jennifer J and Perona, Pietro and Anderson, David J and Kennedy, Ann},
citation = {eLife 2021;10:e63720},
abstract = {The study of naturalistic social behavior requires quantification of animals' interactions. This is generally done through manual annotation---a highly time-consuming and tedious process. Recent advances in computer vision enable tracking the pose (posture) of freely behaving animals. However, automatically and accurately classifying complex social behaviors remains technically challenging. We introduce the Mouse Action Recognition System (MARS), an automated pipeline for pose estimation and behavior quantification in pairs of freely interacting mice. We compare MARS's annotations to human annotations and find that MARS's pose estimation and behavior classification achieve human-level performance. We also release the pose and annotation datasets used to train MARS to serve as community benchmarks and resources. Finally, we introduce the Behavior Ensemble and Neural Trajectory Observatory (BENTO), a graphical user interface for analysis of multimodal neuroscience datasets. Together, MARS and BENTO provide an end-to-end pipeline for behavior data extraction and analysis in a package that is user-friendly and easily modifiable.},
article_type = {journal},
author = {Segalin, Cristina and Williams, Jalani and Karigo, Tomomi and Hui, May and Zelikowsky, Moriel and Sun, Jennifer J and Perona, Pietro and Anderson, David J and Kennedy, Ann},
citation = {eLife 2021;10:e63720},
date-modified = {2022-10-05 13:18:49 -0400},
doi = {10.7554/eLife.63720},
editor = {Berman, Gordon J and Wassum, Kate M and Gal, Asaf},
issn = {2050-084X},
journal = {eLife},
keywords = {social behavior, pose estimation, machine learning, computer vision, microendoscopic imaging, software},
month = {nov},
pages = {e63720},
pub_date = {2021-11-30},
publisher = {eLife Sciences Publications, Ltd},
title = {The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice},
url = {https://doi.org/10.7554/eLife.63720},
volume = 10,
year = 2021,
bdsk-url-1 = {https://doi.org/10.7554/eLife.63720}}
doi = {10.7554/eLife.63720},
editor = {Berman, Gordon J and Wassum, Kate M and Gal, Asaf},
issn = {2050-084X},
journal = {eLife},
keywords = {social behavior, pose estimation, machine learning, computer vision, microendoscopic imaging, software},
month = {nov},
pages = {e63720},
pub_date = {2021-11-30},
publisher = {eLife Sciences Publications, Ltd},
title = {The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice},
url = {https://doi.org/10.7554/eLife.63720},
volume = 10,
year = 2021,
bdsk-url-1 = {https://doi.org/10.7554/eLife.63720}
}


@misc{rastermap,
author = {C. Stringer and M. Pachitariu},
title = {rastermap},
year = {2020},
author = {C. Stringer and M. Pachitariu},
title = {rastermap},
year = {2020},
publisher = {GitHub},
journal = {GitHub repository},
url = {https://github.com/MouseLand/rastermap}
journal = {GitHub repository},
url = {https://github.com/MouseLand/rastermap}
}

@misc{vidio,
author = {J. Bohnslav},
title = {VidIO: simple, performant video reading and writing in python},
year = {2020},
author = {J. Bohnslav},
title = {VidIO: simple, performant video reading and writing in python},
year = {2020},
publisher = {GitHub},
journal = {GitHub repository},
url = {https://github.com/jbohnslav/vidio}
journal = {GitHub repository},
url = {https://github.com/jbohnslav/vidio}
}

@misc{umap,
doi = {10.48550/ARXIV.1802.03426},
url = {https://arxiv.org/abs/1802.03426},
author = {McInnes, Leland and Healy, John and Melville, James},
keywords = {Machine Learning (stat.ML), Computational Geometry (cs.CG), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction},
doi = {10.48550/ARXIV.1802.03426},
url = {https://arxiv.org/abs/1802.03426},
author = {McInnes, Leland and Healy, John and Melville, James},
keywords = {Machine Learning (stat.ML), Computational Geometry (cs.CG), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction},
publisher = {arXiv},
year = {2018},
year = {2018},
copyright = {arXiv.org perpetual, non-exclusive license}
}

@misc{petrucco_2020_3925903,
author = {Petrucco, Luigi},
title = {Mouse head schema},
month = jul,
year = 2020,
publisher = {Zenodo},
doi = {10.5281/zenodo.3925903},
url = {https://doi.org/10.5281/zenodo.3925903}
}
author = {Petrucco, Luigi},
title = {Mouse head schema},
month = jul,
year = 2020,
publisher = {Zenodo},
doi = {10.5281/zenodo.3925903},
url = {https://doi.org/10.5281/zenodo.3925903}
}


@article{NWB,
article_type = {journal},
title = {The Neurodata Without Borders ecosystem for neurophysiological data science},
author = {Rübel, Oliver and Tritt, Andrew and Ly, Ryan and Dichter, Benjamin K and Ghosh, Satrajit and Niu, Lawrence and Baker, Pamela and Soltesz, Ivan and Ng, Lydia and Svoboda, Karel and Frank, Loren and Bouchard, Kristofer E},
editor = {Colgin, Laura L and Jadhav, Shantanu P},
volume = 11,
year = 2022,
month = {oct},
pub_date = {2022-10-04},
pages = {e78362},
citation = {eLife 2022;11:e78362},
doi = {10.7554/eLife.78362},
url = {https://doi.org/10.7554/eLife.78362},
abstract = {The neurophysiology of cells and tissues are monitored electrophysiologically and optically in diverse experiments and species, ranging from flies to humans. Understanding the brain requires integration of data across this diversity, and thus these data must be findable, accessible, interoperable, and reusable (FAIR). This requires a standard language for data and metadata that can coevolve with neuroscience. We describe design and implementation principles for a language for neurophysiology data. Our open-source software (Neurodata Without Borders, NWB) defines and modularizes the interdependent, yet separable, components of a data language. We demonstrate NWB’s impact through unified description of neurophysiology data across diverse modalities and species. NWB exists in an ecosystem, which includes data management, analysis, visualization, and archive tools. Thus, the NWB data language enables reproduction, interchange, and reuse of diverse neurophysiology data. More broadly, the design principles of NWB are generally applicable to enhance discovery across biology through data FAIRness.},
keywords = {Neurophysiology, data ecosystem, data language, data standard, FAIR data, archive},
journal = {eLife},
issn = {2050-084X},
publisher = {eLife Sciences Publications, Ltd}
}
6 changes: 4 additions & 2 deletions paper.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,20 +34,22 @@ Direct inspection of behavior and neurophysiology recordings is hard because the

We provide dedicated widgets and loading functions for exploring raw video, 3D animal pose, behavior annotations, electrophysiology recordings, and calcium imaging data - either as a raster or as a super-position of labeled regions of interest (ROIs). More broadly, SNUB can dislay any data that takes the form of a heatmap, scatter plot, video, or collection of named temporally-varying signals.

In addition to the front-end GUI, we include a library of functions for ingesting raw data and saving it to a format that is readable by the SNUB viewer. The following code, for example, creates a project with paired electrophysiology and video data.
In addition to the front-end GUI, we include a library of functions that ingest raw data and save it to a format that is readable by the SNUB viewer. The following code, for example, creates a project with paired electrophysiology and video data.

```
snub.io.create_project(project_directory, duration=1800)
snub.io.add_video(project_directory, 'path/to/my_video.avi', name='IR_camera')
snub.io.add_splikeplot(project_directory, 'my_ephys_data', spike_times, spike_labels)
```

We also provide a rudimentary tool for automatically generating SNUB datasets from Neurodata Without Borders (NWB) files, which contain raw and processed data from neuroscience recordings [@NWB]. The data in NWB files are stored hierarchically, and each component of the hierarchy has a specific neurodata type that reflects the measurement modality (e.g, "Units" for spike trains, "ImageSeries" for video). Our conversion tool generates a SNUB display element for each supported neurodata type. Users can optionally restrict this process to a subset of the NWB hierarchy (e.g., include pose tracking while excluding electrophysiology, or include just a subset of electrophysiology measurements).

SNUB is a flexible general-purpose tool that complements more specialized packages such as rastermap [@rastermap] and Bento [@bento]. The rastermap interface, for example, is hard-coded for the display of neural activity rasters, ROIs and 2D embeddings of neural activity. Bento is hard-coded for the display of neural activity rasters, behavioral videos and behavioral annotations. SNUB can reproduce either of these configurations and is especially useful when one wishes to include additional types of data or more directly customize the way that data is rendered.

The graphics in SNUB are powered by vispy [@vispy]. SNUB includes wrappers for several dimensionality reduction methods, including rastermap [@rastermap] for ordering raster plots and UMAP [@umap] for 2D scatter plots. Fast video loading is enabled by vidio [@vidio]. The app icon was adapted from a drawing contributed to scidraw by Luigi Petrucco [@petrucco_2020_3925903].

# Acknowledgements

We are grateful to Mohammed Osman for initial contributions to the 3D keypoint visualization tool. CW is a Fellow of The Jane Coffin Childs Memorial Fund for Medical Research. SRD is supported by NIH grants U19NS113201, RF1AG073625, R01NS114020, the Brain Research Foundation, and the Simons Collaboration on the Global Brain.
We are grateful to Mohammed Osman for contributions to the 3D keypoint visualization and NWB conversion tools. CW is a Fellow of The Jane Coffin Childs Memorial Fund for Medical Research. SRD is supported by NIH grants U19NS113201, RF1AG073625, R01NS114020, the Brain Research Foundation, and the Simons Collaboration on the Global Brain.

# References

0 comments on commit 41c1614

Please sign in to comment.