Overlay segmentations

[JuanPabloMontoya271]

Enabled segmentation overlay for NIFTI, NRRD, and VASP.

Currently it only supports single label segmentations loaded from the GUI Inspector.

Next steps:

• Add multi label segmentations
• Add Segmentation Transfer Function to modify the colors of the labels
• Add Dicom segmentations

Data retrieved from TotalSegmentator Dataset

[mlavik1]

Hi!
Thanks a lot for this PR. I'll see if I can test it sometime this weekend :)

Update: Might need a few more days (these weeks have been a bit busy)

[MichaelOvens]

Segmentation overlay would be a great addition to this project, but I'm not sure about the VolumeLoader class - it seems much thicker than it needs to be. There are already established functions and classes for loading in volumes - why do we need a new one? It's also unclear from the naming that this doesn't just load volumes, it loads segmentations.

I'm wondering if a better approach would be to load the segmentation volume as a separate dataset, then expose an inspector field on the Volume Rendered Object where it could be dropped and a button pressed to apply the dataset as a segmentation, which would copy the segmentation's _DataTex to the base object's _SegmentationText and then destroy the segmentation object. The advantage would be that we can re-use all of the existing import pathways; the disadvantage would be that it would require rendering the segmentation before applying it. But that might not be a bad thing as it would allow visual inspection of the segmentation before application.

[mlavik1]

Hi @JuanPabloMontoya271 and @MichaelOvens
Thanks to both of you for your contributions here (code, suggestions, etc.).

I think what @MichaelOvens suggested could work well. I've made a prototype implementation, and plan to finalise that and make a PR soon. That can also be used for PET/CT datasets, such as this one: https://drive.google.com/file/d/1FOmLZGQBVLp66DcDYAjWs5qxoW0zo30f/view?usp=sharing
(source: https://www.aliza-dicom-viewer.com/download/datasets)

I could let the user load two datasets, and then link them together. Then in the shader I'll have a secondary volume and secondary transfer function. Later we could also implement multiple transfer functions for each segmentation label. And as an optimisation, it might be possible to combine the two datasets together in one texture, to make texture fetch faster. Lots of fun things to work on!

Questions:

• Do you have any extra sample datasets that you could share with me? I'll try out the TotalSegmentator dataset as suggested in the PR. Other suggestions are welcome too!
• If you have any examples form other software that I could use as a reference, I'd greatly appreciate any suggestions!

[mlavik1]

Hi again, So I've made a PR based on what we discussed above: #264
I needed to implement support for PET/CT rendering, and it turned out that can be implemented quite similarly to segmentations. So the implementation supports both PET/CT and segmentations. It uses a secondary dataset and transfer function. For PET/CT the TF will map density values to colour/alpha. For segmentations the TF is internally used to map segmentation label ID to colour.

There are also two render modes for segmentations:

• Overlay: Draw segmentations on top of volume
• Isolate: Only render the part of the volume that overlaps with a segmentation label

Next I'll see if I can add a feature that lets you set a seperate TF for each segmentation label. That should be fairly easy, since we can just stack the TFs on top of one another in the generated Texture2D.

Thanks to both of you for suggestions!
I'll likely merge this soon, but let me know if you notice anything that looks wrong, or if you have any further suggestions.

[JuanPabloMontoya271]

Thank you, that sounds like a good plan. I had also been experimenting with PET/CT, it will be a good addition.

Let me know if there is something I can help with.

[mlavik1]

Thank you, that sounds like a good plan. I had also been experimenting with PET/CT, it will be a good addition.

Let me know if there is something I can help with.

Ok, that's nice! You might have more experience with that than me then, since this was my first experiment with PET/CT. So please let me know if I've overlooked something important.
And thanks, you've already helped a lot! (both with the initial implementation here and the dataset you recommended).
I'll happily add you to the CREDITS.md, if you don't mind? UPDATE: I see you did that already in this PR, so I just added your name to the CREDITS.md in my branch as well :)

[JuanPabloMontoya271]

Thank you for adding me to the credits :)