hippunfold simple use case returns error

[sheyma]

Hi all,

having the following data structure:

test_data/
└── sub-100307/
    └── anat/
        ├── sub-100307_T1w.nii.gz
        └── sub-100307_T1wT2w.nii.gz

I have called this command:

$ hippunfold test_data test_out participant --modality T1w --cores 8 --use-singularity

which resulted in following error:

MissingInputException: Missing input files for rule copy_to_results:
work/sub-100307/surf/sub-100307_hemi-L_space-T1w_den-0p5mm_label-hipp_surfaces.spec
WorkflowError:
    WorkflowError:
        MissingInputException: Missing input files for rule copy_to_results:
        work/sub-100307/surf/sub-100307_hemi-L_space-T1w_den-0p5mm_label-hipp_surfarea.shape.gii
        CyclicGraphException: Cyclic dependency on rule warp_gii_to_native.
    MissingInputException: Missing input files for rule copy_to_results:
    work/sub-100307/surf/sub-100307_hemi-L_space-T1w_den-0p5mm_label-hipp_gyrification.shape.gii

It looks like hippunfold complains about missing *gii files.

The error was identical while trying to run the test data set.

Can anyone else reproduce this error, ie. using the test data set?

Provided is my python environment laplac, at which hippunfold was installed as instructions recommend:

conda activate laplac
git clone https://github.com/khanlab/hippunfold.git
pip install hippunfold/

laplac.txt

[sheyma]

So, python=3.8 and hippunfold==1.0.1rc2

[akhanf]

What is the output you get if you run the following (from within the hippunfold directory):

hippunfold test_data/bids_T1w test_out participant --modality T1w --dry-run

It should just output all the rules it will run, but not actually run anything. Let me know if you get an error there.

Note that the test data here: https://github.com/khanlab/hippunfold/tree/master/test_data
is just for dry-run only, ie they are zero-sized niftis, so if you run without the -n or --dry-run option it will fail.

[sheyma]

it still gives the same error with --dry-run option on the test_data, searching for some *gii files and cannot find.

$ hippunfold bids_T1w test_out participant --modality T1w --dry-run

Config file config/snakebids.yml is extended by additional config specified via the command line.
Building DAG of jobs...
WorkflowError:
MissingInputException: Missing input files for rule copy_to_results:
work/sub-001/surf/sub-001_hemi-L_space-T1w_den-0p5mm_label-hipp_surfaces.spec
WorkflowError:
    MissingInputException: Missing input files for rule copy_to_results:
    work/sub-001/surf/sub-001_hemi-L_space-T1w_den-0p5mm_label-hipp_gyrification.shape.gii
    WorkflowError:
        MissingInputException: Missing input files for rule copy_to_results:
        work/sub-001/surf/sub-001_hemi-L_space-T1w_den-0p5mm_label-hipp_surfarea.shape.gii
        CyclicGraphException: Cyclic dependency on rule warp_gii_to_native.

[sheyma]

@jordandekraker : Did you also get the same error while running it on the test_data ?

@akhanf : It could be that you are running hippunfold with singularity or docker, so maybe that's why you don't get this error.

[akhanf]

Hmm if the dry run isn't working then some required file from the workflow is missing or can't be found. The dry run should work even if not using singulairty/docker. The only difference in my dev environment from yours is I am using poetry to install instead of pip install, but both should work (conda/pip is used in the dockerfile).

Can you paste the full output from the dry run with --debug-dag?

Also, did you run hippunfold_download_models first?

[sheyma]

Nope, I did not run hippunfold_download_models before, running it now.
It looks like my home dir will be super full soon, do you know how I could assign a directory for the download destination?

I tried this:

$ echo $HIPPUNFOLD_CACHE_DIR
/data/p_02542/dekrak/.cache/hippunfold/

But it still says that HIPPUNFOLD_CACHE_DIR is not defined.

 $ hippunfold_download_models
HIPPUNFOLD_CACHE_DIR not defined, using default location
Downloading T1w model...
   url = https://zenodo.org/record/4508747/files/trained_model.3d_fullres.Task101_hcp1200_T1w.nnUNetTrainerV2.model_best.tar?dl=1
   dest = /data/hu_bayrak/.cache/hippunfold/trained_model.3d_fullres.Task101_hcp1200_T1w.nnUNetTrainerV2.model_best.tar

[akhanf]

Ah yes I see now that the current instructions don't state this is needed, sorry we need to update that! This is likely the root of the error.

Re: it not finding the env var, that is strange, could something be resetting your environment?

One alternative to try:

HIPPUNFOLD_CACHE_DIR=/data/p_02542/dekrak/.cache/hippunfold/ hippunfold_download_models

[sheyma]

perfect, the above trick looks to be downloading it to the desired destination.

[jordandekraker]

I think for future calls to hippunfold to recognize this cachedir, you'll want to run
export HIPPUNFOLD_CACHE_DIR=/data/p_02542/dekrak/.cache/hippunfold/
similarly, you may want to run something like this for the later steps inside hippunfold involving singularity:

export SINGULARITY_CACHEDIR=/data/p_02542/dekrak/.cache/singularity
export SINGULARITY_BINDPATH=/data:/data

I'll add a bit of text on this in the Documentation's Contributors section.

[sheyma]

download with hippunfold_download_models was successful.
HIPPUNFOLD_CACHE_DIR is now exported.
dry run on test_data was also successful: hippunfold bids_T1w test_out participant --modality T1w --dry-run
... now going to run it with my data.

[sheyma]

Alright, the command below was not successful running on my data:

$ hippunfold test_data test_out participant --modality T1w --cores 8 --use-singularity
Here is the log file:
2022-02-09T184907.566293.snakemake.log

[sheyma]

I also included the debug option as @akhanf was asking earlier:
$ hippunfold test_data test_out participant --modality T1w --cores 8 --use-singularity --debug-dag
2022-02-09T190252.666003.snakemake.log

[jordandekraker]

are you running on MacOS by any chance? This looks like the same error from #165

[akhanf]

Some stderr output might not make it into the snakemake log - but it looks like maybe the run_inference step is failing?
Can you provide the contents of that log?
logs/sub-100307/sub-100307_hemi-R_space-corobl_nnunet.txt

[sheyma]

@jordandekraker : it is all running on Ubuntu

bayrak@comps06h03~ $ lsb_release -a
LSB Version:	core-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.6 LTS
Release:	18.04
Codename:	bionic

and comps0603 has 400 GB ram.

@akhanf : attached is the log file. Apparently a python package torch is missing. I will try to load it and give another try.
sub-100307_hemi-R_space-corobl_nnunet.txt

[akhanf]

Looks like torch is installed for you (looking at your python environment), so not sure why you are getting that error, but likely some conflict in dependencies. Given the problems you're having with setting up the environment, I recommend you run from the hippunfold singularity container - either with the instructions on the docs, or if you want to be able to edit the workflow you can do this:

singularity exec -e docker://khanlab/hippunfold:latest <path_to_cloned_repo>/hippunfold/run.py

The other possible issues could GPU/CUDA related (it may be the issue you are currently facing), and could be avoided by disabling the GPU when you run hippunfold: CUDA_VISIBLE_DEVICES="" or if using singularity: SINGULARITYENV_CUDA_VISIBLE_DEVICES="" (the inference is fast enough without GPU anyhow)

[sheyma]

Looks like torch is installed for you (looking at your python environment)

this was indeed not true anymore, since I had re-generated my conda environment laplac at some point, which was missing torch.

Now, torch package is installed and it looks all running. torch is probably dealing with GPU/CUDA related issues.

... will report the results here after (hopefully) a successful run.

[sheyma]

Alright, all worked out.

As a summary:

conda create --name laplac python=3.8
conda activate laplac
conda install -c pytorch pytorch 
git clone https://github.com/khanlab/hippunfold.git
pip install hippunfold/
export HIPPUNFOLD_CACHE_DIR=/data/p_02542/dekrak/.cache/hippunfold/
hippunfold_download_models
hippunfold test_data test_out participant --modality T1w --cores 8 --use-singularity 

thanks a lot for the step-by-step debugging here! It was amazing to get it run with @akhanf & @jordandekraker

[jordandekraker]

Looks great!
Thanks for taking the time to work through this, I'll be sure to update the Documentation to hopefully avoid confusion in the future!

[sheyma]

all with pleasure :)