NiftyMIC is a Python-based open-source toolkit for research developed within the GIFT-Surg project to reconstruct an isotropic, high-resolution volume from multiple, possibly motion-corrupted, stacks of low-resolution 2D slices. The framework relies on slice-to-volume registration algorithms for motion correction and reconstruction-based Super-Resolution (SR) techniques for the volumetric reconstruction.
The algorithm and software were developed by Michael Ebner at the Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London (UCL) (2015 -- 2019), and the Department of Surgical and Interventional Sciences, King's College London (KCL) (since 2019).
A detailed description of the NiftyMIC algorithm is found in EbnerWang2020:
- Ebner, M., Wang, G., Li, W., Aertsen, M., Patel, P. A., Aughwane, R., Melbourne, A., Doel, T., Dymarkowski, S., De Coppi, P., David, A. L., Deprest, J., Ourselin, S., Vercauteren, T. (2020). An automated framework for localization, segmentation and super-resolution reconstruction of fetal brain MRI. NeuroImage, 206, 116324.
A later extension to this paper for fetal brain automatic segmentation building on MONAI was provided by Marta B.M. Ranzini.
More information about MONAIfbs can be found here.
An extension of NiftyMIC for fetal functional MRI was developed in collaboration with members of the Computational Image Research Lab, Department of Biomedical Imaging and Image-guided Therapy at the Medical University of Vienna. A detailed description of the NiftyMIC algorithm for fetal fMRI is found in Sobotka2022:
- Sobotka, D., Ebner, M., Schwartz, E., Nenning, K.-H., Taymourtash, A., Vercauteren, T., Ourselin, S., Kasprian, G., Prayer, D., Langs, G., Licandro, R. (2022). Motion Correction and Volumetric Reconstruction for Fetal fMRI. arXiv.
If you have any questions or comments, please drop an email to [email protected].
Given a set of low-resolution, possibly motion-corrupted, stacks of 2D slices, NiftyMIC produces an isotropic, high-resolution 3D volume. As an example, we illustrate its use for fetal MRI by computing a high-resolution visualization of the brain for a neck mass subject. Standard clinical HASTE sequences were used to acquire the low-resolution images in multiple orientations.
The associated brain masks for motion correction can be obtained with the included automated segmentation tool MONAIfbs (a legacy method for automated segmentation can also be found in the fetal_brain_seg package).
Full working examples on automated segmentation and high-resolution reconstruction of fetal brain MRI using NiftyMIC is described in the Usage section below.
Figure 1. NiftyMIC -- a volumetric MRI reconstruction tool based on rigid slice-to-volume registration and outlier-robust super-resolution reconstruction steps -- applied to fetal brain MRI.
Figure 2. Qualitative comparison of the original low-resolution input data and the obtained high-resolution volumetric reconstructions in both the original patient-specific and standard anatomical orientations. Five input stacks (two axial, one coronal and two sagittal) were used.
Several methods have been implemented to solve the Robust Super-Resolution Reconstruction (SRR) problem
to obtain the (vectorized) high-resolution 3D MRI volume from multiple, possibly motion corrupted, low-resolution stacks of (vectorized) 2D MR slices
with
for
for a variety of regularizers
and data loss functions
.
The linear operator
represents the combined operator describing the (rigid) motion
, the blurring operator
and the downsampling operator
.
The toolkit relies on an iterative motion-correction/reconstruction approach whereby complete outlier rejection of misregistered slices is achieved by iteratively solving the SRR problem for a slice-index set
containing only slices that are in high agreement with their simulated counterparts projected from a previous high-resolution iterate
according to a similarity measure
and parameter
. In the current implementation, the similarity measure
corresponds to Normalized Cross Correlation (NCC).
The provided data loss functions are motivated by SciPy and allow for additional robust outlier handling during the SRR step. Implemented data loss functions are:
linear:soft_l1:huber:arctan:cauchy:
The available regularizers include
- Zeroth-order Tikhonov (TK0):
- First-order Tikhonov (TK1):
- Isotropic Total Variation (TV):
- Huber Function:
Additionally, the choice of finding optimal reconstruction parameters is facilitated by the Numerical Solver Library (NSoL).
NiftyMIC supports medical image registration and volumetric reconstruction for ultra-fast 2D MRI. NiftyMIC is not intended for clinical use.
If you use this software in your work for structural MRI reconstruction, please cite
- [EbnerWang2020] Ebner, M., Wang, G., Li, W., Aertsen, M., Patel, P. A., Aughwane, R., Melbourne, A., Doel, T., Dymarkowski, S., De Coppi, P., David, A. L., Deprest, J., Ourselin, S., Vercauteren, T. (2020). An automated framework for localization, segmentation and super-resolution reconstruction of fetal brain MRI. NeuroImage, 206, 116324.
- [EbnerWang2018] Ebner, M., Wang, G., Li, W., Aertsen, M., Patel, P. A., Melbourne, A., Doel, T., David, A. L., Deprest, J., Ourselin, S., Vercauteren, T. (2018). An Automated Localization, Segmentation and Reconstruction Framework for Fetal Brain MRI. In Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2018 (pp. 313–320). Springer.
- [Ebner2018] Ebner, M., Chung, K. K., Prados, F., Cardoso, M. J., Chard, D. T., Vercauteren, T., Ourselin, S. (2018). Volumetric reconstruction from printed films: Enabling 30 year longitudinal analysis in MR neuroimaging. NeuroImage, 165, 238–250.
If you use this software in your work for functional MRI reconstruction, please cite
- [Sobotka2022] Sobotka, D., Ebner, M., Schwartz, E., Nenning, K.-H., Taymourtash, A., Vercauteren, T., Ourselin, S., Kasprian, G., Prayer, D., Langs, G., Licandro, R. (2022). Motion Correction and Volumetric Reconstruction for Fetal fMRI. arXiv.
- [EbnerWang2020] Ebner, M., Wang, G., Li, W., Aertsen, M., Patel, P. A., Aughwane, R., Melbourne, A., Doel, T., Dymarkowski, S., De Coppi, P., David, A. L., Deprest, J., Ourselin, S., Vercauteren, T. (2020). An automated framework for localization, segmentation and super-resolution reconstruction of fetal brain MRI. NeuroImage, 206, 116324.
NiftyMIC was developed in Ubuntu 16.04, 18.04 and Mac OS X 10.12 and tested for Python 2.7, 3.5 and 3.6. It builds on a couple of additional libraries developed within the GIFT-Surg project including
whose installation requirements need to be met. Therefore, the local installation comes in three steps:
- Installation of ITK_NiftyMIC
- Installation of SimpleReg dependencies
- Installation of NiftyMIC (including optional MONAIfbs)
Note: The optional use of the automated fetal brain segmentation tool MONAIfbs requires Python version >= 3.6.
To avoid manual installation from source, NiftyMIC is also available as a virtual machine and Docker image.
Provided the input MR image data in NIfTI format (nii or nii.gz), NiftyMIC can reconstruct an isotropic, high-resolution volume from multiple, possibly motion-corrupted, stacks of low-resolution 2D slices.
A recommended workflow is [associated applications in square brackets]:
- Segmentation of the anatomy of interest for all input images. For fetal brain MRI reconstructions, we recommend the use of the fully automatic segmentation tool MONAIfbs already integrated in NiftyMIC [
niftymic_segment_fetal_brains]. - Bias-field correction [
niftymic_correct_bias_field] - Volumetric reconstruction in subject space using two-step iterative approach based on rigid slice-to-volume registration and SRR cycles [
niftymic_reconstruct_volume]
In case reconstruction in a template space is desired (like for fetal MRI) additional steps could be:
- Register obtained SRR to template and update respective slice motion corrections [
niftymic_register_image] - Volumetric reconstruction in template space [
niftymic_reconstruct_volume_from_slices]
Additional information on how to use NiftyMIC and its applications is provided in the following.
Leveraging a two-step registration-reconstruction approach an isotropic, high-resolution 3D volume can be generated from multiple stacks of low-resolution slices.
An example for a basic usage reads
niftymic_reconstruct_volume \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--filenames-masks path-to-stack-1_mask.nii.gz ... path-to-stack-N_mask.nii.gz \
--output path-to-srr.nii.gz \
whereby complete outlier removal during SRR is activated by default (--outlier-rejection 1).
A more elaborate example could be
niftymic_reconstruct_volume \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--filenames-masks path-to-stack-1_mask.nii.gz ... path-to-stack-N_mask.nii.gz \
--alpha 0.01 \
--outlier-rejection 1 \
--threshold-first 0.5 \
--threshold 0.85 \
--intensity-correction 1 \
--isotropic-resolution 0.8 \
--two-step-cycles 3 \
--output path-to-output-dir/srr.nii.gz \
--subfolder-motion-correction motion_correction \ # created in 'path-to-output-dir'
--verbose 1
The obtained motion-correction transformations in motion_correction can be used for further processing, e.g. by using niftymic_reconstruct_volume_from_slices.py to solve the SRR problem for a variety of different regularization and data loss function types.
If a template is available, it is possible to obtain a SRR in its associated standard anatomical space. Using the subject-space SRR outcome of niftymic_reconstruct_volume a rigid alignment step maps all slice motion correction transformations accordingly using
niftymic_register_image \
--fixed path-to-template.nii.gz \
--fixed-mask path-to-template_mask.nii.gz \
--moving path-to-subject-space-srr.nii.gz \
--moving-mask path-to-subject-space-srr_mask.nii.gz \
--dir-input-mc dir-to-motion_correction \
--output path-to-registration-transform.txt
For fetal brain template space alignment, a spatio-temporal atlas is provided in data/templates. If you make use of it, please cite
- Gholipour, A., Rollins, C. K., Velasco-Annis, C., Ouaalam, A., Akhondi-Asl, A., Afacan, O., Ortinau, C. M., Clancy, S., Limperopoulos, C., Yang, E., Estroff, J. A. & Warfield, S. K. (2017). A normative spatiotemporal MRI atlas of the fetal brain for automatic segmentation and analysis of early brain growth. Scientific Reports 7, 476.
and abide by the license agreement as described in data/templates/LICENSE.
After performed/updated motion correction (or having static data in the first place) several options are available:
- Volumetric reconstruction in template space
- Parameter tuning for SRR:
- different solvers and regularizers can be used to solve the SRR problem for comparison
- parameter studies can be performed to find optimal reconstruction parameters.
Solve the SRR problem for motion corrected data (or static data if --dir-input-mc is omitted):
niftymic_reconstruct_volume_from_slices \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--filenames-masks path-to-stack-1_mask.nii.gz ... path-to-stack-N_mask.nii.gz \
--dir-input-mc dir-to-motion_correction \ # optional
--output path-to-srr.nii.gz \
--reconstruction-type TK1L2 \
--reconstruction-space path-to-template.nii.gz \ # optional
--alpha 0.01
niftymic_reconstruct_volume_from_slices \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--dir-input-mc dir-to-motion_correction \
--output path-to-srr.nii.gz \
--reconstruction-type HuberL2 \
--alpha 0.003
Slices that were rejected during the niftymic_reconstruct_volume run are recognized as outliers based on the content of dir-input-mc and will not be incorporated during the volumetric reconstruction.
The optimal choice for reconstruction parameters like the regularization parameter or data loss function can be found by running parameter studies. This includes L-curve studies and direct comparison against a reference volume for various cost functions. In case a reference is available, similarity measures are evaluated against this "ground-truth" as well.
Example are:
niftymic_run_reconstruction_parameter_study \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--filenames-masks path-to-stack-1_mask.nii.gz ... path-to-stack-N_mask.nii.gz \
--dir-input-mc dir-to-motion_correction \
--dir-output dir-to-param-study-output \
--reconstruction-type TK1L2 \
--reconstruction-space path-to-reconstruction-space.nii.gz \ # define reconstruction space
--alphas 0.005 0.01 0.02 0.05 0.1 # regularization parameters to sweep through
--append # if given, append a previously performed parameter study in output directory (if available)
niftymic_run_reconstruction_parameter_study \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--filenames-masks path-to-stack-1_mask.nii.gz ... path-to-stack-N_mask.nii.gz \
--dir-input-mc dir-to-motion_correction \
--dir-output dir-to-param-study-output \
--reconstruction-type HuberL2 \
--reference path-to-reference-volume.nii.gz \ # in case reference ("ground-truth") is available (reconstruction space is defined by this reference)
--measures MAE RMSE PSNR NCC NMI SSIM \ # evaluate reconstruction similarities against reference
--reference-mask path-to-reference-volume_mask.nii.gz \ # if given, evaluate similarities (--measures) on masked region only
--alphas 0.001 0.003 0.005 0.001 0.003 \ # regularization parameters to sweep through
--append # if given, append a previously performed parameter study in output directory (if available)
The results can be assessed by accessing the NSoL-script show_parameter_study.py via
niftymic_show_parameter_study \
--dir-input dir-to-param-study-output \
--study-name TK1L2 \
--dir-output-figures dir-to-figures
An automated framework is implemented to obtain a high-resolution fetal brain MRI reconstruction in the standard anatomical planes (Figure 2). This includes two main subsequent blocks:
- Automatic segmentation to generate fetal brain masks
- Automatic high-resolution reconstruction.
The latter is based on the work described in EbnerWang2018 and EbnerWang2020. Compared to the segmentation approach proposed in EbnerWang2020, a new automated segmentation tool has been included in the NiftyMIC package, called MONAIfbs. This implements a single-step segmentation approach based on the dynUNet implemented in MONAI.
The current NiftyMIC version is still compatible with the older segmentation pipeline based on the fetal_brain_seg
package and presented in EbnerWang2020. Details on its use are available at this wiki page,
although MONAIfbs is the recommended option.
Provided the dependencies for MONAIfbs are installed, create the automatic fetal brain masks of HASTE-like images:
niftymic_segment_fetal_brains \
--filenames \
nifti/name-of-stack-1.nii.gz \
nifti/name-of-stack-2.nii.gz \
nifti/name-of-stack-N.nii.gz \
--filenames-masks \
seg/name-of-stack-1.nii.gz \
seg/name-of-stack-2.nii.gz \
seg/name-of-stack-N.nii.gz
Afterwards, four consecutive steps including
- bias field correction (
niftymic_correct_bias_field), - subject-space reconstruction (
niftymic_reconstruct_volume), - template-space alignment (
niftymic_register_image), and - template-space reconstruction (
niftymic_reconstruct_volume_from_slices)
are performed to create a high-resolution fetal brain MRI reconstruction in the standard anatomical planes:
niftymic_run_reconstruction_pipeline \
--filenames \
nifti/name-of-stack-1.nii.gz \
nifti/name-of-stack-2.nii.gz \
nifti/name-of-stack-N.nii.gz \
--filenames-masks \
seg/name-of-stack-1.nii.gz \
seg/name-of-stack-2.nii.gz \
seg/name-of-stack-N.nii.gz \
--dir-output srr
Additional parameters such as the regularization parameter alpha can be specified too. For more information please execute niftymic_run_reconstruction_pipeline -h.
Note: In case a suffix distinguishes image segmentation (--filenames-masks) from the associated image filenames (--filenames), the argument --suffix-mask needs to be provided for reconstructing the HR brain volume mask as part of the pipeline. E.g. if images name-of-stack-i.nii.gz are associated with the mask name-of-stack-i_mask.nii.gz, then the additional argument --suffix-mask _mask needs to be specified.
An extension of NiftyMIC for fetal functional MRI is described in Sobotka2022:
Figure 3: Overview of the proposed motion correction and volumetric reconstruction algorithm for rs-fMRI. With the first n=15 (default) time points a HR reference volume with outlier rejection is estimated (Ebner, Wang et al., 2020). Afterwards each slice of a time point is registered to the HR reference volume following which individual time points are reconstructed using Huber L2 regularization.
Figure 4: Qualitative comparison of the original low-resolution input sequence (top row) and the obtained volumetric reconstruction of the functional MRI (bottom row) in patient-specific orientation. A sequence of axial stacks were used as input.
A recommended workflow is [associated applications in square brackets]:
-
Volumetric segmentation of the fetal brain for all input images in the sequence. We tested the algorithm with fetal functional brains masks created using the Brain Extraction Tool incorporated in the FSL Toolbox (Smith et al. 2002).
-
Motion estimation [
rsfmri_estimate_motion]rsfmri_estimate_motion \ --filename path-to-bold.nii.gz \ --filename-mask path-to-bold_mask.nii.gz \ --dir-output path-to-dir-output -
Functional volumetric reconstruction [
rsfmri_reconstruct_volume_from_slices]An example for a basic usage reads
rsfmri_reconstruct_volume_from_slices \ --filename path-to-bold.nii.gz \ --filename-mask path-to-bold_mask.nii.gz \ --dir-input-mc path-to-motion_correction \ # created in `dir-output` in step 2 above --output path-to-bold_reconstructed.nii.gzA more elaborate example using a different reconstruction approach
HuberL2could bersfmri_reconstruct_volume_from_slices \ --filename path-to-bold.nii.gz \ --filename-mask path-to-bold_mask.nii.gz \ --dir-input-mc path-to-motion_correction \ # created in `dir-output` in step 2 above --output path-to-bold_reconstructed.nii.gz \ --reconstruction-type HuberL2 --alpha 0.1 \ --iter-max 20
If you have any questions or comments with regards to the use of NiftyMIC for functional MRI, please drop an email to [email protected].
Copyright (c) 2022 Michael Ebner and contributors. This framework is made available as free open-source software under the BSD-3-Clause License. Other licenses may apply for dependencies.
This work is partially funded by the UCL Engineering and Physical Sciences Research Council (EPSRC) Centre for Doctoral Training in Medical Imaging (EP/L016478/1), the Innovative Engineering for Health award (Wellcome Trust [WT101957] and EPSRC [NS/A000027/1]), and supported by researchers at the National Institute for Health Research University College London Hospitals (UCLH) Biomedical Research Centre.
Selected publications associated with NiftyMIC are:
- [Sobotka2022] Sobotka, D., Ebner, M., Schwartz, E., Nenning, K.-H., Taymourtash, A., Vercauteren, T., Ourselin, S., Kasprian, G., Prayer, D., Langs, G., Licandro, R. (2022). Motion Correction and Volumetric Reconstruction for Fetal fMRI. arXiv.
- [EbnerWang2020] Ebner, M., Wang, G., Li, W., Aertsen, M., Patel, P. A., Aughwane, R., Melbourne, A., Doel, T., Dymarkowski, S., De Coppi, P., David, A. L., Deprest, J., Ourselin, S., Vercauteren, T. (2020). An automated framework for localization, segmentation and super-resolution reconstruction of fetal brain MRI. NeuroImage, 206, 116324.
- [Ebner2019] Ebner, M., Patel, P. A., Atkinson, D., Caselton, C., Firmin, F., Amin, Z., Bainbridge, A., De Coppi, P., Taylor, S. A., Ourselin, S., Chouhan, M. D., Vercauteren, T. (2019). Super‐resolution for upper abdominal MRI: Acquisition and post‐processing protocol optimization using brain MRI control data and expert reader validation. Magnetic Resonance in Medicine, 82(5), 1905–1919.
- [Sobotka2019] Sobotka, D., Licandro, R., Ebner, M., Schwartz, E., Vercauteren, T., Ourselin, S., Kasprian, G., Prayer, D., Langs, G. (2019). Reproducibility of Functional Connectivity Estimates in Motion Corrected Fetal fMRI. Smart Ultrasound Imaging and Perinatal, Preterm and Paediatric Image Analysis (pp. 123–132). Cham: Springer International Publishing.
- [EbnerWang2018] Ebner, M., Wang, G., Li, W., Aertsen, M., Patel, P. A., Melbourne, A., Doel, T., David, A. L., Deprest, J., Ourselin, S., Vercauteren, T. (2018). An Automated Localization, Segmentation and Reconstruction Framework for Fetal Brain MRI. In Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2018 (pp. 313–320). Springer
- [Ebner2018] Ebner, M., Chung, K. K., Prados, F., Cardoso, M. J., Chard, D. T., Vercauteren, T., Ourselin, S. (2018). Volumetric reconstruction from printed films: Enabling 30 year longitudinal analysis in MR neuroimaging. NeuroImage, 165, 238–250.
- [Ranzini2017] Ranzini, M. B., Ebner, M., Cardoso, M. J., Fotiadou, A., Vercauteren, T., Henckel, J., Hart, A., Ourselin, S., and Modat, M. (2017). Joint Multimodal Segmentation of Clinical CT and MR from Hip Arthroplasty Patients. MICCAI Workshop on Computational Methods and Clinical Applications in Musculoskeletal Imaging (MSKI) 2017.
- [Ebner2017] Ebner, M., Chouhan, M., Patel, P. A., Atkinson, D., Amin, Z., Read, S., Punwani, S., Taylor, S., Vercauteren, T., Ourselin, S. (2017). Point-Spread-Function-Aware Slice-to-Volume Registration: Application to Upper Abdominal MRI Super-Resolution. In Zuluaga, M. A., Bhatia, K., Kainz, B., Moghari, M. H., and Pace, D. F., editors, Reconstruction, Segmentation, and Analysis of Medical Images. RAMBO 2016, volume 10129 of Lecture Notes in Computer Science, pages 3–13. Springer International Publishing.