From 8b4d16f08d42a8429ed07294635ad9e30efc1e70 Mon Sep 17 00:00:00 2001 From: Kyle Meyer Date: Tue, 12 Jun 2018 15:13:54 -0400 Subject: [PATCH 1/6] reproin: Add missing link for datalad uninstall --- _episodes/03-01-reproin.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_episodes/03-01-reproin.md b/_episodes/03-01-reproin.md index bb9802c..ba61730 100644 --- a/_episodes/03-01-reproin.md +++ b/_episodes/03-01-reproin.md @@ -513,6 +513,7 @@ TODO metadata [datalad siblings]: http://datalad.readthedocs.io/en/latest/generated/man/datalad-siblings.html [datalad sshrun]: http://datalad.readthedocs.io/en/latest/generated/man/datalad-sshrun.html [datalad subdatasets]: http://datalad.readthedocs.io/en/latest/generated/man/datalad-subdatasets.html +[datalad uninstall]: http://datalad.readthedocs.io/en/latest/generated/man/datalad-uninstall.html [datalad update]: http://datalad.readthedocs.io/en/latest/generated/man/datalad-update.html [datalad containers-add]: http://docs.datalad.org/projects/container/en/latest/generated/man/datalad-containers-add.html [datalad containers-list]: http://docs.datalad.org/projects/container/en/latest/generated/man/datalad-containers-list.html From b836dcc2999ffa5e2c022611d80c4560c34a28f6 Mon Sep 17 00:00:00 2001 From: Kyle Meyer Date: Tue, 12 Jun 2018 15:13:54 -0400 Subject: [PATCH 2/6] reproin: Update script links These were moved in ohbm2018-training's 2026209 (reorganize, 2018-06-10). --- _episodes/03-01-reproin.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/_episodes/03-01-reproin.md b/_episodes/03-01-reproin.md index ba61730..4869042 100644 --- a/_episodes/03-01-reproin.md +++ b/_episodes/03-01-reproin.md @@ -344,9 +344,9 @@ Before we can fire up FSL for our GLM analysis, we need two pieces of custom code: 1. a small script that can convert BIDS events.tsv files into the EV3 format that - FSL can understand: available at + FSL can understand: available at -2. an FSL analysis configuration template script available at: +2. an FSL analysis configuration template script available at: Any custom code needs to be tracked if we want to achieve a complete record of how an analysis was conducted. Hence we will store those scripts in our analysis @@ -360,8 +360,8 @@ dataset. > > > ## Solution > > ~~~ -> > % datalad download-url --path code/events2ev3.sh https://raw.githubusercontent.com/myyoda/ohbm2018-training/master/scripts/events2ev3.sh -> > % datalad download-url --path code/ffa_design.fsf https://raw.githubusercontent.com/myyoda/ohbm2018-training/master/scripts/ffa_design.fsf +> > % datalad download-url --path code/events2ev3.sh https://raw.githubusercontent.com/myyoda/ohbm2018-training/master/section23/scripts/events2ev3.sh +> > % datalad download-url --path code/ffa_design.fsf https://raw.githubusercontent.com/myyoda/ohbm2018-training/master/section23/scripts/ffa_design.fsf > > % git log > > > > ~~~ From aeff62460391045f3e88aabdc547ac486b190d03 Mon Sep 17 00:00:00 2001 From: Kyle Meyer Date: Tue, 12 Jun 2018 15:13:54 -0400 Subject: [PATCH 3/6] reproin: Revise (uncontroversial typos/grammar) --- _episodes/03-01-reproin.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_episodes/03-01-reproin.md b/_episodes/03-01-reproin.md index 4869042..7bc7b50 100644 --- a/_episodes/03-01-reproin.md +++ b/_episodes/03-01-reproin.md @@ -25,7 +25,7 @@ Importantly, we will conduct this analysis so that it: for improved reproducibility and portability For all steps of this study, we will use [DataLad] to achieve these goals with -relatively minimal effort +relatively minimal effort. > #### DataLad Extensions @@ -231,7 +231,7 @@ information. > that you can use in the command specification itself. > > Use `git log` to investigate what information [DataLad] captured about this -> command executions. +> command's execution. > > > ## Solution > > Prefix the original command line call with @@ -317,7 +317,7 @@ Regarding the layout of this analysis dataset, we unfortunately cannot yet rely on automatic tools and a comprehensive standard (but such guidelines are actively being worked on). However, DataLad nevertheless aids efforts to bring order to the chaos. Anyone can develop their own ideas on how a dataset should -be structured, and implement these concepts in *dataset procedures* that can be +be structured and implement these concepts in *dataset procedures* that can be executed using the [datalad run-procedure] command. Here we are going to adopt the YODA principles: a set of simple rules on how to @@ -453,7 +453,7 @@ to complete on an average system. feat {inputs[0]} > ~~~ -Once this command finished, DataLad will have captured the entire FSL output, +Once this command finishes, DataLad will have captured the entire FSL output, and the dataset will contain a complete record all the way from the input BIDS dataset to the GLM results (which, by the way, performed an FFA localization on a real BOLD imaging dataset, take a look!). The BIDS subdataset in turn has a @@ -470,7 +470,7 @@ And because this record is complete, we can now simply throw away the input BIDS > ## Task: Verify that the BIDS subdataset is unmodified and uninstall it > > Use the [datalad diff] command to check for modifications of the subdataset, -> and the [datalad uninstall] do delete it. +> and the [datalad uninstall] to delete it. > > > ## Solution > > ~~~ From d2214d104ba099463aa64f64119548cefc395ed4 Mon Sep 17 00:00:00 2001 From: Kyle Meyer Date: Tue, 12 Jun 2018 15:13:54 -0400 Subject: [PATCH 4/6] reproin: Revise (minor stylistic changes) --- _episodes/03-01-reproin.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/_episodes/03-01-reproin.md b/_episodes/03-01-reproin.md index 7bc7b50..60f31d6 100644 --- a/_episodes/03-01-reproin.md +++ b/_episodes/03-01-reproin.md @@ -14,12 +14,12 @@ keypoints: In this lesson, we will carry out a full (although very basic) functional imaging study, going from raw data to complete data-analysis results. We will start from imaging data in DICOM format — as if we had just finished scanning. -Importantly, we will conduct this analysis so that it: +Importantly, we will conduct this analysis so that it - leaves a comprehensive "paper trail" of all performed steps; everything will be tracked via version control -- structures components in a way that facilitates re-use +- structures components in a way that facilitates reuse - performs all critical computation in containerized computational environments for improved reproducibility and portability @@ -93,7 +93,7 @@ directory so that all further commands will be able to use relative paths. When using DataLad, it is best to always run scripts from the root directory of the dataset — and also code all scripts to use paths that are relative to this root directory. For this to work, a dataset must contain all of the inputs of a -processing step (all code; all data). +processing step (all code, all data). That means that we should add the raw DICOM files to our BIDS dataset. In our case, these DICOMs are already available in a DataLad dataset from @@ -242,6 +242,7 @@ information. > > --input inputs/rawdata/events.tsv \ > > --output sub-02/func/sub-02_task-oneback_run-01_events.tsv \ > > cp {inputs} {outputs} +> > % git log -p > > ~~~ > > {: .bash} > {: .solution} @@ -344,9 +345,9 @@ Before we can fire up FSL for our GLM analysis, we need two pieces of custom code: 1. a small script that can convert BIDS events.tsv files into the EV3 format that - FSL can understand: available at + FSL can understand, available at -2. an FSL analysis configuration template script available at: +2. an FSL analysis configuration template script available at Any custom code needs to be tracked if we want to achieve a complete record of how an analysis was conducted. Hence we will store those scripts in our analysis @@ -401,7 +402,7 @@ expects. First, let's convert the events.tsv file into EV3 format files. All we have left is to configure the desired first-level GLM analysis with FSL. The following command will create a working configuration from the template we stored in `code/`. It uses the arcane, yet powerful `sed` editor. We will again -use [datalad run] to invoke our command, so that we store in the history +use [datalad run] to invoke our command so that we store in the history *how* this template was generated (so that we may audit, alter, or regenerate this file in the future — fearlessly). @@ -436,7 +437,7 @@ shub://mih/ohbm2018-training:fsl {: .challenge} The command we will run is a simple `feat sub-02/1stlvl_design.fsf`. However, in -order to achieve the most reproducible and most portable execution we should +order to achieve the most reproducible and most portable execution, we should tell the [datalad containers-run] command what the inputs and outputs are. DataLad will then be able to obtain the required NIfTI time series file from the BIDS raw subdataset. From 2030dddbd3a620c3627af3f311c4eebfea207bc7 Mon Sep 17 00:00:00 2001 From: Kyle Meyer Date: Tue, 12 Jun 2018 15:13:54 -0400 Subject: [PATCH 5/6] reproin: Don't hardcode HEAD~10 in afterlife solution Using HEAD~10 is fragile. It needs to be updated if any of the steps above are modified, and it assumes the solutions above were achieved in exactly the same way, but it would be possible to get to the same state with a different number of commits (e.g., by using 'datalad download-url --path code/'). --- _episodes/03-01-reproin.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/_episodes/03-01-reproin.md b/_episodes/03-01-reproin.md index 60f31d6..a96ee5e 100644 --- a/_episodes/03-01-reproin.md +++ b/_episodes/03-01-reproin.md @@ -470,12 +470,14 @@ And because this record is complete, we can now simply throw away the input BIDS > ## Task: Verify that the BIDS subdataset is unmodified and uninstall it > -> Use the [datalad diff] command to check for modifications of the subdataset, -> and the [datalad uninstall] to delete it. +> Use the [datalad diff] command and `git log` to verify that the subdataset is +> in the same state as when it was initially added. Then use [datalad +> uninstall] to delete it. > > > ## Solution > > ~~~ -> > % datalad diff --revision HEAD~10 -- inputs +> > % datalad diff -- inputs +> > % git log -- inputs > > % datalad uninstall --dataset . inputs --recursive > > ~~~ > > {: .bash} From 06b634c2b2f41d967d37d289f7f0ae2dec1f346f Mon Sep 17 00:00:00 2001 From: Kyle Meyer Date: Tue, 12 Jun 2018 15:13:54 -0400 Subject: [PATCH 6/6] reproin: Simplify download-url call --path accepts a directory and will use the basename from the URL. Note: This commits assumes the previous commit which removes "HEAD~10" from the afterlife solution; otherwise, that solution will fail with this change because it reduces the number of commits in the repo. --- _episodes/03-01-reproin.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/_episodes/03-01-reproin.md b/_episodes/03-01-reproin.md index a96ee5e..110fe6a 100644 --- a/_episodes/03-01-reproin.md +++ b/_episodes/03-01-reproin.md @@ -361,8 +361,9 @@ dataset. > > > ## Solution > > ~~~ -> > % datalad download-url --path code/events2ev3.sh https://raw.githubusercontent.com/myyoda/ohbm2018-training/master/section23/scripts/events2ev3.sh -> > % datalad download-url --path code/ffa_design.fsf https://raw.githubusercontent.com/myyoda/ohbm2018-training/master/section23/scripts/ffa_design.fsf +> > % datalad download-url --path code \ +> > https://raw.githubusercontent.com/myyoda/ohbm2018-training/master/section23/scripts/events2ev3.sh \ +> > https://raw.githubusercontent.com/myyoda/ohbm2018-training/master/section23/scripts/ffa_design.fsf > > % git log > > > > ~~~