Error during bb_fix feature extraction, possibly related to settings

[dsodums]

Dear TVB users,

When running some Rockland data through the pipeline, it is getting stuck on the FIX stage at the extract features stage so I'm hoping there is something that needs editing in a settings file as I was able to run this command without any problems when running it on a different server (RRI). Note that the training file being used was created on the RRI server, but I made sure to change the version date so it can be used by the version of R being used by the singularity container.

This is the command that is currently failing when submitting the entire pipeline:

/home/soddev/projects/def-levine/soddev/Rockland/TVBukbb_pipeline/tvb-ukbb/bb_functional_pipeline/bb_fix_dir/fix . /home/soddev/projects/def-levine/soddev/Rockland/TVBukbb_pipeline/tvb-ukbb/bb_functional_pipeline/bb_fix_dir/training_files/Training_Rockland.RData 30 -m -h 100

Running this command on its own in an interactive session gives me the following output:

FIX Feature extraction for Melodic output directory: .
 create edge masks
 run FAST
 registration of standard space masks
 extract features
FIX Classifying components in Melodic directory: . using training file: /home/soddev/projects/def-levine/soddev/Rockland/TVBukbb_pipeline/tvb-ukbb/bb_functional_pipeline/bb_fix_dir/training_files/Training_Rockland.RData and threshold 30
FIX features not already generated, so creating them now.
FIX Feature extraction for Melodic output directory: .
 create edge masks
 run FAST
 registration of standard space masks
 extract features
No valid labelling file specified

When digging into the output logs, /scratch/soddev/Rockland/Raw_BIDS/sub-A00038158/fMRI/rfMRI.ica/.fix_2b_predict.log has the following output and I'm wondering if something needs to be be changed in the settings (perhaps something to do with the fact that only [1] is defined and the script is then looking for the values defined under the "else" section of the python script:

Loading required package: gplots

Attaching package: 'gplots'

The following object is masked from 'package:stats':

    lowess

> library(randomForest)
randomForest 4.6-12
Type rfNews() to see new features/changes/bug fixes.
> library(MASS)
>
> # R CMD BATCH "--no-restore --args ${FIXDIR}/ ${MELOUT}/ Standard 0.1" ${FIXDIR}/fix_2_PREDICT.R ${MELOUT}/fix/logR_Standard_0.1.txt
>
> if (T) {
+       args <- commandArgs(TRUE)
+       path.to.fix <- args[[1]]
+       print(path.to.fix)
+       melodic.path <- args[[2]]
+       print(melodic.path)
+       weight.file.name <- args[[3]]
+       print(weight.file.name)
+       bin.thresh <- args[[4]]
+       print(bin.thresh)
+       #
+       #which.fix <- args[[5]]
+       #print(which.fix)
+ } else {
+       rm(list = ls())
+       path.to.fix <- "/Users/reza/Documents/Academic/FIX/fix1.02"
+       melodic.path <- "/Users/reza/Documents/Academic/FIX/raw_data/EXAMPLEL/20100509_151441IM-0015-140s015a001.ica"
+       weight.file.name <- "/Users/reza/Documents/Academic/FIX/raw_data/EXAMPLEL/TRAIN.RData"
+       bin.thresh <- 20
+ }
[1] "/home/soddev/projects/def-levine/soddev/Rockland/TVBukbb_pipeline/tvb-ukbb/bb_functional_pipeline/bb_fix_dir"
[1] "."
[1] "/home/soddev/projects/def-levine/soddev/Rockland/TVBukbb_pipeline/tvb-ukbb/bb_functional_pipeline/bb_fix_dir/training_files/Training_Rockland.RData"
[1] "30"
>
>
>
>
>
> if(substr(path.to.fix, nchar(path.to.fix), nchar(path.to.fix))!="/")
+       path.to.fix <- paste(path.to.fix, "/", sep = "")
> if(substr(melodic.path, nchar(melodic.path), nchar(melodic.path))!="/")
+       melodic.path <- paste(melodic.path, "/", sep = "")
> bin.thresh <- as.numeric(bin.thresh)
>
> load(weight.file.name)
> test.data <- read.csv(paste(melodic.path, "fix/features.csv", sep = ""), header = F)
Error in file(file, "rt") : cannot open the connection
Calls: read.csv -> read.table -> file
In addition: Warning message:
In file(file, "rt") :
  cannot open file './fix/features.csv': No such file or directory
Execution halted

The other error log is related to matlab, when looking at this log file:
/scratch/soddev/Rockland/Raw_BIDS/sub-A00038158/fMRI/rfMRI.ica/fix/logMatlab.txt

------------------------------------------
Setting up environment variables
---
LD_LIBRARY_PATH is /usr/lib/x86_64-linux-gnu:/opt/mcr/v93/runtime/glnxa64:/opt/mcr/v93/bin/glnxa64:/opt/mcr/v93/sys/os/glnxa64:/opt/mcr/v93/sys/opengl/lib/glnxa64:/opt/miniconda3/lib:/usr/lib::/.singularity.d/libs
Picked up JAVA_TOOL_OPTIONS: -Xmx2g
/bin/sh -c '. /opt/fsl/etc/fslconf/fsl.sh; FSLOUTPUTTYPE=NIFTI_PAIR; export FSLOUTPUTTYPE; $FSLDIR/bin/fslmaths ./fix/hr2exf /localscratch/soddev.28899167.0/tp6ed8d53a_6d31_4834_b7ea_001871a7562b': Aborted
Error Time: 03/14/2022 10:40:23
Error using call_fsl (line 36)
FSL call (/bin/sh -c '. /opt/fsl/etc/fslconf/fsl.sh; FSLOUTPUTTYPE=NIFTI_PAIR; export FSLOUTPUTTYPE; $FSLDIR/bin/fslmaths ./fix/hr2exf /localscratch/soddev.28899167.0/tp6ed8d53a_6d31_4834_b7ea_001871a7562b') failed, terminate called after throwing an instance of 'NiftiIO::NiftiException'
  what():  Error: cant open file /localscratch/soddev.28899167.0/tp6ed8d53a_6d31_4834_b7ea_001871a7562b.hdr
/bin/sh: line 1: 28526 Aborted                 (core dumped) $FSLDIR/bin/fslmaths ./fix/hr2exf /localscratch/soddev.28899167.0/tp6ed8d53a_6d31_4834_b7ea_001871a7562b

Thanks in advance,
Devin

[noahfl]

Hi Devin!

Looks like the first issue is definitely a problem either with the generation of the MELODIC outputs or with the feature extraction process. It looks like FIX fails the first time to extract the features into fix/features.csv and tries again after checking for fix/features.csv prior to classification. Is fix/features.csv present when fix_2b_predict runs?

For the second issue, I think that's an easy fix. It looks like that MATLAB runtime is looking for /localscratch but it's not visible to the container. Try adding -B /localscratch to your singularity command (following the format of the other -B args) and this one should be resolved.

[dsodums]

Hi Noah!

Thank you very much, the second issue was causing the problem with fix generating the features. Adding -B /localscratch to the singularity command worked like a charm and now the features.csv is created. Out of curiosity, do I need to have both -B /localscratch and -B /scratch when running the singularity command?

[noahfl]

Just saw this! Sorry for the late reply, I didn't get a notification for your response for some reason.

Oh perfect! Glad that sorted itself out 🙂

You don't have to, but there's no harm in doing so. I usually use both to avoid occasional system usages of /localscratch and because I'm running subjects out of /scratch anyway.

[dsodums]

Not a worry at all! That makes sense, thanks for confirming :).