KeyError: 'Name'

[algebio]

• imctools version: 2
• Python version: 3.7
• Operating System: Windows 10

Description

Describe what you were trying to get done.
Tell us what happened, what went wrong, and what you expected to happen.

I'm just running the Jupyter notebook imc_preprocessing.ipynb. I get the same error in two different computers using the same dataset. I have analysed this dataset with imctools v1 without getting any error message.

What I Did

Paste the command(s) you ran and the output.

%%time
for fol in folder_path_ome.iterdir():
    if fol.is_dir():
        ome2histocat.omefolder_to_histocatfolder(fol, folder_path_histocat)


If there was a crash, please include the traceback here.

KeyError Traceback (most recent call last)
in

~\AppData\Local\anaconda3\envs\ImcSegmentationPipeline\lib\site-packages\imctools\converters\ome2histocat.py in omefolder_to_histocatfolder(input_folder, output_folder, mask_folder, mask_suffix, dtype)
119 mask_file = None
120 path_ome = os.path.join(input_folder, fn_ome)
--> 121 omefile_to_histocatfolder(path_ome, output_folder, mask_file=mask_file, dtype=dtype)
122
123

~\AppData\Local\anaconda3\envs\ImcSegmentationPipeline\lib\site-packages\imctools\converters\ome2histocat.py in omefile_to_histocatfolder(filepath, base_folder, mask_file, dtype)
65 basename = filepath.name.rstrip(".ome.tiff")
66 output_folder = base_folder / basename
---> 67 omefile_to_tifffolder(filepath, output_folder, basename="", dtype=dtype)
68 if mask_file is not None:
69 if isinstance(mask_file, str):

~\AppData\Local\anaconda3\envs\ImcSegmentationPipeline\lib\site-packages\imctools\converters\ome2histocat.py in omefile_to_tifffolder(filepath, output_folder, basename, dtype)
33 output_folder.mkdir(parents=True, exist_ok=True)
34
---> 35 with OmeTiffParser(filepath) as parser:
36 acquisition_data = parser.get_acquisition_data()
37 acquisition_data.save_tiffs(output_folder, basename=basename, dtype=dtype)

~\AppData\Local\anaconda3\envs\ImcSegmentationPipeline\lib\site-packages\imctools\io\ometiff\ometiffparser.py in init(self, filepath, slide_id, channel_id_offset)
21 self._slide_id = slide_id
22 self._channel_id_offset = channel_id_offset
---> 23 self._acquisition_data = self._parse_acquisition(filepath)
24
25 @Property

~\AppData\Local\anaconda3\envs\ImcSegmentationPipeline\lib\site-packages\imctools\io\ometiff\ometiffparser.py in _parse_acquisition(self, filepath)
56 def _parse_acquisition(self, filepath: Path):
57 image_data, ome_xml = OmeTiffParser._read_file(filepath)
---> 58 image_name, channel_names, channel_labels, self._mcd_xml = OmeTiffParser._parse_ome_xml(ome_xml)
59
60 max_x = image_data.shape[2]

~\AppData\Local\anaconda3\envs\ImcSegmentationPipeline\lib\site-packages\imctools\io\ometiff\ometiffparser.py in _parse_ome_xml(xml)
104
105 channels = img.findall(f"{ns}Pixels/{ns}Channel")
--> 106 chan_dict = {
107 int(chan.attrib["ID"].split(":")[2]): (chan.attrib["Name"], chan.attrib["Fluor"]) for chan in channels
108 }

~\AppData\Local\anaconda3\envs\ImcSegmentationPipeline\lib\site-packages\imctools\io\ometiff\ometiffparser.py in (.0)
105 channels = img.findall(f"{ns}Pixels/{ns}Channel")
106 chan_dict = {
--> 107 int(chan.attrib["ID"].split(":")[2]): (chan.attrib["Name"], chan.attrib["Fluor"]) for chan in channels
108 }
109

KeyError: 'Name'

[votti]

Hey @algebio
Thanks a lot for the bug report!

This seems to be the same issue that is causing: BodenmillerGroup/ImcSegmentationPipeline#62

@plankter Could you look into this?

[votti]

Hi,

An update: I just freshly installed everything and run tests and cannot see that anything looks suspicious.

Would you be able to share an example .ome.tiff ?

[votti]

How did you generate the .ome.tiff? From .mcd files via imctools? imctools v1 or v2?

[algebio]

Hi @votti

Thanks for your help!

I created the ome.tiff from a .mcd. You can find everything in this folder. I used imctools v2 which I installed it yesterday from GitHub. I also updated Anaconda before running the Jupyter notebook. Something that I noticed when updating the external dependencies is that some of them never got updated and remained in the "need to be updated" list. However, imagecodecs, pandas, xmltodict, xtiff updated fine.

I used the same .mcd a few days ago with the previous version of Anaconda and imctools v1 and I got everything without any error messages.

Regards
Juan

[votti]

Thanks @algebio!!
Having this data was extremly helpful.
Unfortunately I am not longer working in the mass cytometry field, so I so input from the community is really required to keep things runnig.

To the issue:

@plankter : The issue seems to because the ChannelLabel attribute that is used to fill the Name can be empty in the mcd .

This is a legit behavior, as the 'antibody name' is merely a label of the channel. The unique channel identifier is allways the isotope.

In imctools v1 I set the Isotope channel name as label, if the label was not in the mcd:

imctools/imctools/io/mcdxmlparser.py

Line 187 in 8983b97

channel_label = chan.properties.get(CHANNELLABEL, channel_name)

This is helpful as many OME tiff viewers will show the Name (=Channel Label), so having this missing can be confusing/cause issues.

I think this filling-in is not longer done in imctools v2.

There are two t things to do:

1. Make sure the downstream processing also works with ome.tiff where the Name channel attribute is not set
   -> This is crucial to support also ome.tiff generated with imctools so far

Implementation: The quick fix would be to change line:

imctools/imctools/io/ometiff/ometiffparser.py

Line 107 in 00a3242

int(chan.attrib["ID"].split(":")[2]): (chan.attrib["Name"], chan.attrib["Fluor"]) for chan in channels

 int(chan.attrib["ID"].split(":")[2]): (chan.attrib["Name"], chan.attrib["Fluor"]) for chan in channels

to

 int(chan.attrib["ID"].split(":")[2]):(chan.attrib.get("Name",  chan.attrib["Fluor"]), chan.attrib["Fluor"]) for chan in channels

2. (Optional) Bring back the previous behavior to fill the Name attribute with the Channel name in cases that the Channel Label is left empty.

@plankter : Does this make sense? Could you implement this?

@nilseling : I heard you may be doing some acquisitions for a tiny test dataset: This issue could be reproduced by adding a channel manually to the acquisition and not setting a label. Would it be feasible to add this to your test acquisition??

Thanks team for helping to keep this running!

[algebio]

Hi @votti

I'm glad that I could help improve this already great pipeline. I can't wait to see these new changes implemented and use imctools v2 with my dataset.

Regards
Juan

[nilseling]

Hey @votti, I will manually add Ag to capture the counterstain as one of the channels. This can then be used to test Label free channels. To me it would also make sense to bring back the old behaviour and fill the Name attribute with the metal isotope.

[algebio]

Hi

Thank you for addressing this issue. Could you let me know when it is sorted?

Regards
Juan

[plankter]

Hi,

I shall release an update today.

Best regards,
Anton Rau

[plankter]

Thank you @votti for the suggestions!

Version 2.1.5 with the fix is released.

Best,
Anton

[algebio]

Beautiful!

Thank you Vito, Anton and Nils. Great job!

Regards
Juan