More metadata?

[joaomamede]

Hi, when I print the metadata from Nd2Reader I have a lot of things missing (even comparing to the metadata when opening with bioformats).

Is there any way for us to get more metadata with nd2reader?

Example of metadata in the .nd2 files that are not available (by order of preference and need):
EmissionWavelength
The setup PhysicalSizeZ (we can estimate from the Z positions, however I have the accuracy in my experiments to "open" and it varies a bit).
RefractiveIndex
Detector ID /Model etc.

DeltaT (when there's only one T the acquisition time)
ExposureTime,
Position X, Y and Z
Color Set for each Channel in NIS-elements.

Thank you for the module!

[joaomamede]

I realized this is a duplicate. Sorry

[rbnvrw]

Hi @joaomamede there is more metadata available but it is "hidden", there is not a nice parser available yet that converts it from the raw bytes to the proper Python dtypes. Some additional data is available via this class https://github.com/rbnvrw/nd2reader/blob/master/nd2reader/raw_metadata.py which is accessible as reader.parser._raw_metadata. Hope this brings you a bit closer to what you need.

[joaomamede]

hi,
Basically I have my Z step accuracy set to open to speed up live cell imaging. That means that the Z_coordinates are not precisely what the Z distance was set-up for my experiments.
What I mean is, it would be helpful to have the set-up Z-stepping of the experiment. So far I was doing this as a work-around:
scalez = round(nd2meta['z_coordinates'][1]-nd2meta['z_coordinates'][0],3) Which it doesn't always work depending on what that the microscope did.

I ended up setting it manually depending on the experiment. (as I'm deconvolving data and generating PSFs, it's one of the last fields that I can't autogenerate). If I'm doing it it's fine, but I am trying to make a pipeline for other people in the lab (basically a "click and go".

• As you extract the z_coordinates, would the X and Y would be in the same position/style in the nd2? to then append to the metadata?

• Bioformats for example is able to see some of this metadata while not using Nikon's SDK. I would go try to find it, but I would have no idea how to do file byte reading and file parsing.
  For example, PhysicalSizeZ is correctly identified.
  Same as the "color" for visual set in NIS-elements, channel name (this one Nd2Reader does) and the emissionWavelength.
  

Any hints on how I could get incorporated?

These ones I could find:
-(This one is already supported): I saw from the nd2reader source code that we can get the Delta Ts with reader.get_timesteps() (my deltaTs vary from 6minutes in the first 5 hours to 15minutes the following 5 and finally 45 minutes for the rest of the experiment).

• frames.parser._raw_metadata.camera_exposure_time Has the exposure time

EDIT1:
I found the Zstep by doing:
frames.parser._raw_metadata.image_metadata[b'SLxExperiment'][b'ppNextLevelEx'][b''][b'ppNextLevelEx'][b''][b'uLoopPars'][b'dZStep']
Any way I can get this in an easier way?

Thanks so much again.

[rbnvrw]

The approach you mention via the _raw_metadata is indeed the only way to get the "extra" attributes that are not implemented via the .metadata attribute. However, the z levels should also be in the .metadata attribute but I'm not sure if they are the same ones.

I am a bit hesitant to expand the .metadata dictionary because it'll get really big, but one could add a ParsedMetadata class which is accesible via an .complete_metadata attribute on the ND2Reader which would contain all the parsed metadata. I currently don't have the time to implement this but would welcome a PR. I'll leave this open for now as a reminder.

[ggirelli]

Regarding the extraction of the Z step, I have had some adventures with the file generated in my lab.

As the stage is very slightly wobbling every now and then (of approx. 25 nm, as registered by the instrument) it is not obvious to obtain a Z step from the distance between each Z location/coordinate. In my case, I can extract the Z step from the stage metadata by parsing and ImageText field with this regex "Z Stack Loop: ([0-9]+)\r\n- Step: ([0-9,\\.]+) ([^\r\n]*)".

When that fails, I revert to something like what you mentioned:

            Zdata = np.array(parser._raw_metadata.z_data)
            Zlevels = np.array(parser.metadata["z_levels"]).astype("int")
            Zlevels = Zlevels + len(Zlevels) * field_id
            Zdata = Zdata[Zlevels]
            return np.round(np.diff(Zdata), 3).tolist()

And then I check the mode of the calculated Z steps and raise a warning if the mode is less than 50% of the steps (i.e., case of extra-wobbliness of the stage).

Hope this can help.

[joaomamede]

This is what I did

    def fetch_extra_metadata(reader):
        import re
        extra_meta = reader.parser._raw_metadata.image_text_info[b'SLxImageTextInfo'][b'TextInfoItem_5'].decode()
        extra_meta = re.split(',|;|\r\n',extra_meta)
        extra_dict = dict()

        for line in extra_meta:
            line = line.strip().strip('- ')
            keyvalue = str.split(line,':')
            if len(keyvalue) > 1:
                key = keyvalue[0]
                value = keyvalue[1]
                extra_dict[key] = value

        return extra_dict

extra_dict = fetch_extra_metadata(reader)
scalez = extra_dict['Step'].split()[0]

[joaomamede]

"Z Stack Loop: ([0-9]+)\r\n- Step: ([0-9,\.]+) ([^\r\n]*)"

How did you get to this?

My solution above, sometimes fails when the file is a bit different.

image_text_info[b'SLxImageTextInfo'][b'TextInfoItem_5'] is not always there.
(New NIS-elements didn't give me an error so far though)