Accumulator for development outlined in #5

[mih]

There is no need or benefit merging this before it is done. It main purpose is to make the effort transparent.

Resources:

• minimal heudiconv image (161M)
• minimal FSL image (261M)

Images combined are sufficient to go from DICOM to GLM results for the demo functional dataset.

Script is coming.

[satra]

perhaps we can use neurodocker to generate both the docker and singularity containers? or just docker and then use singularity to pull it.

example for your fsl script:

docker run -it --rm kaczmarj/neurodocker:master generate docker \
  --base neurodebian:stretch-non-free \
  --pkg-manager apt \
  --install eatmydata fsl-5.0-core fsl-mni152-templates \
  --run "eatmydata apt-get -y install --no-install-recommends file" \
  --add-to-entrypoint "source /etc/fsl/5.0/fsl.sh"

replace docker with singularity to generate the singularity script.

and correspondingly for heudiconv:

docker run -it --rm kaczmarj/neurodocker:master generate docker \
--base neurodebian:latest \
--pkg-manager apt \
--install eatmydata pigz python3-pip python3-traits python3-scipy \
    python3-setuptools python3-wheel python3-networkx dcm2niix \
--run "pip3 install --system nipype https://github.com/mvdoc/dcmstack/archive/bf/importsys.zip \
   https://github.com/nipy/heudiconv/archive/master.zip"

another reason i say this is because i will be covering neurodocker as a way to create environments in the workflow part.

[mih]

@satra sounds good.

[satra]

@mih - if datalad is setup like this, i.e. itself a container:

alias datalad='singularity run -B $PWD:/datadir -W /datadir datalad.img'

would the current script in this PR work?

[satra]

fixed neurodocker versions:

fsl:

docker run --rm kaczmarj/neurodocker:master generate singularity  \
  --base neurodebian:stretch-non-free   --pkg-manager apt   \
  --install fsl-5.0-core fsl-mni152-templates  \
  --add-to-entrypoint "source /etc/fsl/5.0/fsl.sh" > Singularity

heudiconv:

docker run -i --rm kaczmarj/neurodocker:master generate singularity \
  --base neurodebian:latest --pkg-manager apt \
  --install pigz python3-pip python3-traits python3-scipy   python3-setuptools \
           python3-wheel python3-networkx dcm2niix \
  --run "pip3 install --system nipype https://github.com/mvdoc/dcmstack/archive/bf/importsys.zip https://github.com/nipy/heudiconv/archive/master.zip"

[satra]

@yarikoptic and @mih: should i go ahead and test this script out with the neurodocker containers? are there any changes you are envisioning?

[satra]

heudiconv with reprozip

docker run -i --rm kaczmarj/neurodocker:master generate singularity \
  --base neurodebian:latest --pkg-manager apt \
  --install pigz python3-pip python3-traits python3-scipy  \
     python3-setuptools python3-wheel python3-networkx dcm2niix \
  --install make gcc sqlite3 libsqlite3-dev python3-dev libc6-dev \
  --run "pip3 install --system nipype \
    https://github.com/mvdoc/dcmstack/archive/bf/importsys.zip \ 
    https://github.com/nipy/heudiconv/archive/master.zip \
    reprozip reprounzip" > Singularity

$ sudo singularity build heudiconv.img Singularity
$ singularity run -c -e -B $PWD:/work --pwd /work heudiconv.img reprozip trace ls

fsl with reprozip

docker run --rm kaczmarj/neurodocker:master generate singularity  \
  --base neurodebian:stretch-non-free   --pkg-manager apt   \
  --install fsl-5.0-core fsl-mni152-templates \
  --install make gcc sqlite3 libsqlite3-dev python3-dev \
    libc6-dev python3-pip python3-setuptools python3-wheel \
  --run "pip3 install --system reprozip reprounzip" \
  --add-to-entrypoint "source /etc/fsl/5.0/fsl.sh" > Singularity

$ sudo singularity build fsl.img Singularity
$ singularity run -c -e -B $PWD:/work --pwd /work fsl.img reprozip trace bet
$ singularity run -c -e -B $PWD:/work --pwd /work fsl.img reprozip pack --dont-identify-packages --dont-find-inputs-outputs bet.rpz

[satra]

@mih - testing on system with current datalad master installed.

$ ./fsl_glm_w_amazing_datalad.sh 
Workdir at /tmp/tmp.YrVWe92Cfs/localizerdemo
[INFO   ] Creating a new annex repo at /tmp/tmp.YrVWe92Cfs/bids 
[WARNING] It is highly recommended to configure git first (set both user.name and user.email) before using DataLad. Failed to verify that git is configured: CommandError: command '['git', 'config', 'user.name']' failed with exitcode 1
| Failed to run ['git', 'config', 'user.name'] under None. Exit code=1. out= err= [cmd.py:run:530]CommandError: command '['git', 'config', 'user.email']' failed with exitcode 1
| Failed to run ['git', 'config', 'user.email'] under None. Exit code=1. out= err= [cmd.py:run:530].  Some operations might fail or not perform correctly. 
create(ok): /tmp/tmp.YrVWe92Cfs/bids (dataset)                                  
[ERROR  ] unknown argument: -d. 
usage: datalad install [-h] [-s SOURCE] [-d DATASET] [-g] [-D DESCRIPTION]
                       [-r] [--recursion-limit LEVELS] [--nosave] [--reckless]
                       [-J NJOBS]
                       [PATH [PATH ...]]

[satra]

datalad installed with:

sudo apt-get install git-annex-standalone python3-pip
sudo pip3 install https://github.com/datalad/datalad/archive/master.zip

[satra]

into matt's current virtualbox vm (if other things should be added to configuration - please add to #15)

[mih]

Sorry, the script had a typo from a hand-edit that I didn't fix yet: There cannot be NO space between -d and '.'

I pushed the fix.

[satra]

@mih - next error: datalad: Unknown command 'containers-add'. See 'datalad --help'.

[mih]

sudo pip3 install https://github.com/datalad/datalad-container/archive/master.zip

and anticipating the next error ;-)

sudo pip3 install https://github.com/datalad/datalad-neuroimaging/archive/master.zip

[satra]

@mih - much appreciated. should perhaps add this to section 2/3 here #15 :)

should i just also create a datalad container with singularity inside that runs other containers :) - have you tried this before?

[mih]

We have a container with datalad to run datalad containers. The singularity part of it should work, but you know that time slows dramatically this many level down.

[satra]

The singularity part of it should work, but you know that time slows dramatically this many level down.

also implants many new memories. datalad may never remember the reality!

[satra]

@mih - since all of this takes place in a mktemp directory will datalad retain anything?

[mih]

The tempdir is just for being able to run this script repeatedly as a demo. Look for XXX actual demo from here. The temp-stuff won't be part of the real course material.

[satra]

@mih - merging this for now so i can test a few things.