Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for HiP-CT #1646

Closed
balbasty opened this issue Oct 30, 2023 · 31 comments · Fixed by #1686
Closed

Add support for HiP-CT #1646

balbasty opened this issue Oct 30, 2023 · 31 comments · Fixed by #1686
Labels
BEP enhancement New feature or request

Comments

@balbasty
Copy link
Contributor

Dear BIDS community

Hierarchical Phase-Contrast Tomography (HiP-CT) is an ex vivo X-ray technique that leverages high-energy X-ray sources -- specifically, the ESRF's Extremely Brilliant Source (EBS). HiP-CT is non-destructive, reaches resolutions of 1 to 10 um, and allows specific ROIs to be imaged within the sample without physical sectioning. Although the modality is obviously not widely available, the UCL group has a relatively streamlined acquisition process and we will be collecting data on multiple brains in the coming years.

We would like to add support for HiP-CT in BIDS, and we think it most naturally fits as a new modality suffix within the microscopy extension. This makes sense because micro-CT is already supported as a microscopy suffix. Also, the microscopy extension already includes tags that are required by HiP-CT data (for example, the existence of multiple images of the same samples, potentially with different resolutions and different field-of-views, labeled as chunk-). We are therefore proposing to add the hipCT suffix to the microscopy extension.

We have uploaded an exampled dataset that follows the proposed convention to DANDI:
https://dandiarchive.org/dandiset/000026/draft/files?location=sub-I58%2Fses-Hip-CT%2Fmicr&page=1
We also provide three neuroglancer links that allow each volume to be visualized online, directly streaming the dandi data:

Let me also include a typical metadata file provided by the UCL group:

############# DOI ###################

Title		= "U.N."
Abstract	= "U.N."
Users		= "U.N."


############ proposal ###############

proposalTitle   ="U.N."
proposalNumber  ="md190"
proposers       ="U.N."
experimentType  ="U.N."
technique       ="U.N."
beamlineName    ="BM18"
instrumentName  ="U.N."


############ patient ################

PatientInstitue ="U.N."
PatientNumber   ="I58-LH"
PatientAge      ="U.N. years"
PatientSex      ="U.N."
PatientWeight   ="U.N. kg"
PatientSize     ="U.N. cm"
PatientMedInfo  ="U.N."


########### sample ################

SampleName  ="I58-LH_brain"
SampleInfo  ="complete heart from the body donor program of the Laboratoire d'Anatomie des Alpes Francaise (LADAF)"
SamplePrep  ="formalin fixed, progressive transfer to ethanol 70%. Vacuum degassing at each transfer to new ethanol bath, mounted with agar crushed gel at 70% ethanol"

########### scan parameters ###########

StartTime       ="20/01/2023"
SRCurrent       ="200mA mA"
IDName          ="BM18"
ScanRadix       ="HA-2300_15.13um_I58-LH_brain_C"
XStep           ="U.N."
XStages         ="U.N."
YStep           ="U.N."
YStages         ="U.N."
ZStep           ="5"
ZStages         ="32"
Projections     ="15000"
RefN            ="U.N."
DarkN           ="U.N."
RefOn           ="U.N."
ScanningMode    ="U.N."
ExposureTime    ="0.014 s"
AccExposureTime ="0.07 s"
AccFramesCount  ="5"
PixelSize       ="15.13 um"
PropDistance    ="15000 mm"
Filters         ="Sapphire 10mm SiO2 15mm"
Det_Avg_Energy  ="90 keV"
ScanGeometry    ="half-acquisition"
ScanRange       ="360 deg"
SensorName      ="IRIS15"
SensorMode      ="U.N."
SensorPixelSize ="4.25 um"
Magnification   ="0.3"
XPixelNum       ="5046"
YPixelNum       ="400"
OpticsType      ="Dzoom"
Scintillator    ="LuAG 2000 reflective"
SurfDoseRate    ="26.9 Gy/s"
ScanTime        ="20"
SeriesTime      ="10.7"


######### processing ############

RefApproach     ="U.N."
VolumeX         ="9413"
VolumeY         ="9413"
VolumeZ         ="10656"
32to16BitsMin   ="-0.17"
32to16BitsMax   ="0.31"
Jp2ComprRatio   ="10"

And the corresponding BIDS json file

{
	"IntendedFor":"sub-I58_sample-01_chunk-01_hipCT.ome.json",
	"Manufacturer":"ESRF",
	"StationName": "BM18",
	"InstitutionName":"European Synchrotron Radiation Facility",
	"InstitutionAddress":"71 Av. des Martyrs, 38000 Grenoble, France",
	"PixelSize":"[15.13, 15.13, 15.13]",
	"PixelSizeUnits":"um",
	"OtherAcquisitionParameters": "{\"StartTime\": \"20/01/2023\", \"SRCurrent\": \"200mA mA\", \"IDName\": \"BM18\", \"ScanRadix\": \"HA-2300_15.13um_I58-LH_brain_C\", \"ZStep\": \"5\", \"ZStages\": \"32\", \"Projections\": \"15000\", \"ExposureTime\": \"0.014 s\", \"AccExposureTime\": \"0.07 s\", \"AccFramesCount\": \"5\", \"PixelSize\": \"15.13 um\", \"PropDistance\": \"15000 mm\", \"Filters\": \"Sapphire 10mm SiO2 15mm\", \"Det_Avg_Energy\": \"90 keV\", \"ScanGeometry\": \"half-acquisition\", \"ScanRange\": \"360 deg\", \"SensorName\": \"IRIS15\", \"SensorPixelSize\": \"4.25 um\", \"Magnification\": \"0.3\", \"XPixelNum\": \"5046\", \"YPixelNum\": \"400\", \"OpticsType\": \"Dzoom\", \"Scintillator\": \"LuAG 2000 reflective\", \"SurfDoseRate\": \"26.9 Gy/s\", \"ScanTime\": \"20\", \"SeriesTime\": \"10.7\"}",
	"BodyPart":"Brain",
	"BodyPartDetails":"Left Hemisphere",
	"SampleEnviornment":"ex vivo",
	"SampleEmbedding":"agar crushed gel at 70% ethanol",
	"SampleFixation":"formalin",
	"SampleExtractionInstitution":"Massachusetts General Hospital"
}

Currently, the HiP-CT acquisition parameters are stored in the "OtherAcquisitionParameters" field, as a serialized JSON dictionary. But it may be better to move some of these HiP-CT-specific tags into the BIDS spec, and we would welcome any feedback from users of the other microscopy modalities.

@satra @Kevancic @dstansby

Thanks
Yael
@sappelhoff sappelhoff added enhancement New feature or request BEP labels Oct 30, 2023
@satra satra mentioned this issue Nov 3, 2023
Closed
@Remi-Gau
Copy link
Collaborator

Remi-Gau commented Nov 8, 2023

out of curiosity would it not be an issue if we one day wanted to integrate CT in BIDS:
see https://bids.neuroimaging.io/bep024 ?

@Remi-Gau
Copy link
Collaborator

Remi-Gau commented Nov 8, 2023

an issue in terms of compatibility or ease of use (where should my CT data go?)

@balbasty
Copy link
Contributor Author

balbasty commented Nov 8, 2023

This is a good question.

I am a bit wary of having HiP-CT as a special case of CT for several reasons:

  • It is a phase-based contrast -- not the typical absorption-based contrast used in CT
  • It can only be acquired on ex vivo tissue, which makes it closer to a microscopy technique than an in vivo modality
  • Its very high resolution and ginormous size means that we must use the zarr format, which is only accepted in the standard under micr/, as far as I understand

That said, I am neither a CT nor a HiP-CT expert, let me try to get someone who knows what they are talking about to chime in.

@ucbpclw
Copy link

ucbpclw commented Nov 8, 2023

Hi all, thought I'd jump in here to add some extra info regarding HiP-CT. @balbasty is correct for all the above. I think to summarise and add some of my own reasons to separate HiP-CT from CT:

  1. HiP-CT is ex vivo only
  2. It replies of propagation based phase-contrast
  3. It has a hierarchical resolution format, i.e. the whole organ is imaged at an overview resolution and then specific areas can be imaged at higher resolution creating multi-scale datasets of single organ
  4. As mentioed datasets are large, some of our largest aquired brain data sets are now at 14TB for one sample.
  5. HiP-CT is a synchrotron technique, meaning that the meta-data fields for both acquisition and also reconstruction will be quite different and varied compared with a clinical or even a lab based CT system.

Also there is more detail on the technique in this paper , and you can see some video that give a clearer idea of the technique here.

@Remi-Gau
Copy link
Collaborator

Remi-Gau commented Nov 8, 2023

OK it feels pretty clear from what you are saying that this is quite different from clinical CT.

From what I understand it looks like adding a microscopy suffix is needed to start supporting this.

Currently, the HiP-CT acquisition parameters are stored in the "OtherAcquisitionParameters" field, as a serialized JSON dictionary.

In many cases BIDS lets you add extra metadata, so no need to to have it serialized however better in this case to start listing the metadata that will be REQUIRED for downstream analysis and add those to the bids spec

@satra
Copy link
Collaborator

satra commented Nov 8, 2023

@balbasty - do you want to start a PR to the spec for this?

@Remi-Gau
Copy link
Collaborator

Remi-Gau commented Nov 8, 2023

I would not mind also seeing an example added to the bids example repo: https://github.com/bids-standard/bids-examples#contributing

@satra
Copy link
Collaborator

satra commented Nov 8, 2023

looks like adding a microscopy suffix

you mean under the micr prefix. the suffix is just hipCT as proposed right?

seeing an example added to the bids example

perhaps a downscaled version for the moment could work. may be just at the coarsest levels. one of the the images that @balbasty linked to has 1 million objects :)

@balbasty
Copy link
Contributor Author

balbasty commented Nov 8, 2023

Sure I can draft a PR.

The one thing I am not sure yet is which metadata should be added (if any). @ucbpclw do you think any of the metadata should be REQUIRED or RECOMMENDED to store in the sidecar json file?

And don't worry I won't push terabytes of data to the example repo! Something like the 1mm resolution level would make sense I'd say.

@effigies
Copy link
Collaborator

effigies commented Nov 8, 2023

I don't think we inspect the contents of zarr directories. You could drop everything but any metadata files inside that may need to be inspected to ensure consistency between OME and BIDS.

@ucbpclw
Copy link

ucbpclw commented Nov 9, 2023 via email

@Remi-Gau
Copy link
Collaborator

Remi-Gau commented Nov 9, 2023 •
edited
Loading

to illustrate what @effigies said, here are the content of the microscopy examples:

as you can see we tend to truncate large files to keep the repo light: https://github.com/bids-standard/bids-examples/blob/master/CONTRIBUTING.md#why-do-we-only-host-truncated-data-with-0kb-size

$ tree --si micr_S* 

[4.1k]  micr_SEM
├── [ 157]  dataset_description.json
├── [ 651]  participants.json
├── [  98]  participants.tsv
├── [ 168]  README
├── [ 330]  samples.json
├── [  60]  samples.tsv
└── [4.1k]  sub-01
    ├── [4.1k]  ses-01
    │   └── [4.1k]  micr
    │       ├── [   1]  sub-01_ses-01_sample-A_photo.jpg
    │       ├── [ 137]  sub-01_ses-01_sample-A_photo.json
    │       ├── [ 373]  sub-01_ses-01_sample-A_SEM.json
    │       └── [   1]  sub-01_ses-01_sample-A_SEM.png
    ├── [4.1k]  ses-02
    │   └── [4.1k]  micr
    │       ├── [ 137]  sub-01_ses-02_sample-A_photo.json
    │       ├── [   1]  sub-01_ses-02_sample-A_photo.tif
    │       ├── [ 373]  sub-01_ses-02_sample-A_SEM.json
    │       └── [   1]  sub-01_ses-02_sample-A_SEM.png
    ├── [ 156]  sub-01_sessions.json
    └── [  74]  sub-01_sessions.tsv
[4.1k]  micr_SEMzarr
├── [ 157]  dataset_description.json
├── [ 651]  participants.json
├── [  98]  participants.tsv
├── [ 210]  README
├── [ 330]  samples.json
├── [  60]  samples.tsv
└── [4.1k]  sub-01
    ├── [4.1k]  ses-01
    │   └── [4.1k]  micr
    │       ├── [ 373]  sub-01_ses-01_sample-A_SEM.json
    │       ├── [   1]  sub-01_ses-01_sample-A_SEM.png
    │       ├── [  70]  sub-01_ses-01_sample-A_SPIM.json
    │       └── [4.1k]  sub-01_ses-01_sample-A_SPIM.ome.zarr
    │           └── [4.1k]  0
    │               └── [4.1k]  0
    │                   └── [ 128]  0
    ├── [4.1k]  ses-02
    │   └── [4.1k]  micr
    │       ├── [ 373]  sub-01_ses-02_sample-A_SEM.json
    │       └── [   1]  sub-01_ses-02_sample-A_SEM.png
    ├── [ 156]  sub-01_sessions.json
    └── [  74]  sub-01_sessions.tsv
[4.1k]  micr_SPIM
├── [ 158]  dataset_description.json
├── [ 388]  participants.json
├── [  54]  participants.tsv
├── [ 616]  README
├── [ 330]  samples.json
├── [  83]  samples.tsv
└── [4.1k]  sub-01
    └── [4.1k]  micr
        ├── [ 387]  sub-01_sample-A_photo.json
        ├── [   1]  sub-01_sample-A_photo.png
        ├── [ 635]  sub-01_sample-A_stain-LFB_chunk-01_SPIM.json
        ├── [2.7k]  sub-01_sample-A_stain-LFB_chunk-01_SPIM.ome.tif
        ├── [ 637]  sub-01_sample-A_stain-LFB_chunk-02_SPIM.json
        ├── [2.7k]  sub-01_sample-A_stain-LFB_chunk-02_SPIM.ome.tif
        ├── [ 636]  sub-01_sample-A_stain-LFB_chunk-03_SPIM.json
        ├── [2.7k]  sub-01_sample-A_stain-LFB_chunk-03_SPIM.ome.tif
        ├── [ 635]  sub-01_sample-A_stain-LFB_chunk-04_SPIM.json
        ├── [2.7k]  sub-01_sample-A_stain-LFB_chunk-04_SPIM.ome.tif
        ├── [ 387]  sub-01_sample-B_photo.json
        ├── [   1]  sub-01_sample-B_photo.png
        ├── [ 635]  sub-01_sample-B_stain-LFB_chunk-01_SPIM.json
        ├── [2.7k]  sub-01_sample-B_stain-LFB_chunk-01_SPIM.ome.tif
        ├── [ 637]  sub-01_sample-B_stain-LFB_chunk-02_SPIM.json
        ├── [2.7k]  sub-01_sample-B_stain-LFB_chunk-02_SPIM.ome.tif
        ├── [ 635]  sub-01_sample-B_stain-LFB_chunk-03_SPIM.json
        ├── [2.7k]  sub-01_sample-B_stain-LFB_chunk-03_SPIM.ome.tif
        ├── [ 635]  sub-01_sample-B_stain-LFB_chunk-04_SPIM.json
        └── [2.7k]  sub-01_sample-B_stain-LFB_chunk-04_SPIM.ome.tif

@balbasty
Copy link
Contributor Author

balbasty commented Nov 9, 2023

@ucbpclw

Do you mean which of metadata fields we sent in our example file are essential?

Yes, exactly

@balbasty balbasty mentioned this issue Jan 24, 2024
Merged
@balbasty
Copy link
Contributor Author

balbasty commented Apr 26, 2024 •
edited
Loading

Hi

I am reopening this thread. After further discussions with "synchrotron-CT" experts (@chourroutm, @dstansby, @ucbpclw), it appeared that hipCT might not be the most adequate suffix. The "H" means hierarchical and really relates to the acqusition of multiple images in the same sample. Each individual image is a (very fancy) phase-contrast CT.

They have suggested "PC" for phase-contrast, although in the current spec, "PC" relates to optical phase-contrast, not xray phase-contrast.

It might be good to fix this suffix before the next releast of the spec.

@satra @ayendiki @effigies @yarikoptic

@effigies effigies reopened this Apr 26, 2024
@effigies
Copy link
Collaborator

XPC - X-ray Phase Contrast?

@ayendiki
Copy link

ayendiki commented Apr 26, 2024 via email

@dstansby
Copy link

dstansby commented May 2, 2024

Agree that it's worth dropping the "Hi" of "HiP-CT" - as mentioned above stands for Hierarchical which is about how mutiple datasets/scans relate to each other (basically some are higher resolution VOIs of others).

XPC sounds good. Does the suffix need to indicate it's CT data though? I note there's already a microCT suffix in the spec. So how about something like xPCCT? Or just PCCT for phase contrast computed tomography?

@effigies
Copy link
Collaborator

effigies commented May 2, 2024

XPC sounds good. Does the suffix need to indicate it's CT data though?

The only hard and fast rule is that it needs to be alphanumeric. Otherwise, if the affected community agrees that it's precise enough, and people outside the community don't see it conflicting with something familiar to them, then it's all good.

I note there's already a microCT suffix in the spec. So how about something like xPCCT? Or just PCCT for phase contrast computed tomography?

I don't love xPCCT, but XPC and PCCT seem equally fine to me. But I don't work with these data, so I'm happy to leave it to those who do.

@dstansby
Copy link

dstansby commented May 2, 2024

If there's no particular standard/rules for what has to be in the suffix, apart from understandable by the community, then maybe it's worth keeping this as hipCT?

@chourroutm
Copy link
Contributor

chourroutm commented May 3, 2024 •
edited
Loading

@dstansby I'd rather go for a broader name, as we're the only group/consortium doing HiP-CT (at the moment) but the community using the same underlying technique --- with or without the hierarchical component, which indeed rather corresponds to a dataset rather than a sole volume --- is already quite developed .

The different names I am aware of are:

As there is no real consensus of the naming within the community, we will not please everyone, of course. It seems to me that XPC might be sufficient as a suffix to the microscopy keyword. Otherwise, to convey the 3D, maybe XPCT is the one to go.

@chourroutm
Copy link
Contributor

What's the opinion of the others? @ucbpclw @balbasty @satra
My former collaborators were also interested in such an addition, I can include them in the decision-making.

Then, what is the course of events? Should I draft a PR with one of the two suffices?

I suppose we can prepare in the following months (once the validator recognizes the suffix) a truncated version of the open brain dataset from the Human Organ Atlas: https://human-organ-atlas.esrf.eu/datasets/572252538

@satra
Copy link
Collaborator

satra commented May 7, 2024

@chourroutm - i would leave this to you and the other experts to decide. as long as it doesn't conflict, we should be ok with any choice of a new extension.

@balbasty
Copy link
Contributor Author

balbasty commented May 7, 2024

@chourroutm I vote XPCT, but yes it'd be great to have a validation from other groups, so that it lives up to the test of time :)

Once it's ok'd, you (or I) should revert the changes from PR #1686 and replace all occurences of hipCT by XPCT.

@effigies
Copy link
Collaborator

effigies commented May 7, 2024

It would be good to get an example in https://github.com/bids-standard/bids-examples as well, so that we can update and test the validator. See the contributing guide.

@ucbpclw
Copy link

ucbpclw commented May 7, 2024

I would vote XPCT as well, it is the wording we are generally using when writing grants as it captures all the essential components of the technique and is specific enough to differentiate it from other X-ray or tomography examples.

@chourroutm chourroutm mentioned this issue May 15, 2024
Merged
@chourroutm
Copy link
Contributor

chourroutm commented May 15, 2024 •
edited
Loading

@GobeJ and @DoctorEmmetBrown do you want to weigh in?

@chourroutm
Copy link
Contributor

@effigies I've tailored an example dataset in chourroutm/bids-xpct-example/tree/master/micr_XPCTzarr that we'll amend before I draft a PR to bids-standard/bids-examples.

@ucbpclw @dstansby and eventually @balbasty can I have your inputs on the dataset? This is the file structure of the example dataset:

micr_XPCTzarr
|   dataset_description.json
|   participants.json
|   participants.tsv
|   README.md
|   samples.json
|   samples.tsv
|   
+---code
|       data_retrieval.py
|       data_retrieval_reqs.txt
|       metadata_formatter.py
|       metadata_formatter_reqs.txt
|       truncate_data.sh
|       
\---sub-LADAF-2020-31
    |   sub-LADAF-2020-31_sessions.json
    |   sub-LADAF-2020-31_sessions.tsv
    |   
    \---ses-01
        \---micr
            |   sub-LADAF-2020-31_ses-01_sample-brain_photo.jpg
            |   sub-LADAF-2020-31_ses-01_sample-brain_photo.json
            |   sub-LADAF-2020-31_ses-01_sample-brain_XPCT.json
            |   
            \---sub-LADAF-2020-31_ses-01_sample-brain_XPCT.ome.zarr
                |   .zattrs
                |   .zgroup
                |   
                +---0
                |   |   .zarray
                |   [...]
                [...]
                |
                +---4
                    |        
                    [...]
                    |           
                    \---9
                        +---0
                        |       0
                        |       1
                        |       2
                        |       
                        +---1
                        |       0
                        |       1
                        |       2
                        |       
                        \---2
                                0
                                1
                                2

@dstansby
Copy link

Layout looks good to me (FWIW I sucessfully adapted a HiP-CT dataset to BIDS to upload to LINC), but I think that's beyond the scope of this issue (which is just about the prefix)?

@chourroutm chourroutm mentioned this issue May 22, 2024
Merged
@effigies
Copy link
Collaborator

#1822 changing hipCT to XPCT has been open a week with no objections. I would ask someone else to leave an approving review and encourage anybody who would object to do so as soon as possible.

@ucbpclw ucbpclw mentioned this issue May 23, 2024
Open
@fabienchauveau
Copy link

I am also in favour of "XPCT".

@effigies
Copy link
Collaborator

XPCT adopted in #1822.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
BEP enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ENH] Add support for HiP-CT balbasty/bids-specification
10 participants
@effigies @satra @dstansby @Remi-Gau @balbasty @sappelhoff @ucbpclw @ayendiki @chourroutm @fabienchauveau