Error when opening OME-Zarr created with bioformats2raw #72

tischi · 2022-02-17T12:37:02Z

Plugins › BigDataViewer › OME ZARR › Open OME ZARR From File System...

with

/g/cba/exchange/ome-zarr/2022/lifdata1__bf2raw-0-4-0.zarr

gives

(Fiji Is Just) ImageJ 2.3.0/1.53f51; Java 1.8.0_202 [64-bit]; Mac OS X 10.14.6; 586MB of 8953MB (6%)
 
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
	at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:307)
	at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166)
	at ij.IJ.runPlugIn(IJ.java)
	at ij.Executer.runCommand(Executer.java:150)
	at ij.Executer.run(Executer.java:68)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:303)
	... 5 more
Caused by: java.lang.RuntimeException: Module threw exception
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
Caused by: java.lang.UnsupportedOperationException: Could not find multiscales at 
	at org.embl.mobie.io.ome.zarr.loaders.N5OMEZarrImageLoader.getMultiscale(N5OMEZarrImageLoader.java:279)
	at org.embl.mobie.io.ome.zarr.loaders.N5OMEZarrImageLoader.initSetups(N5OMEZarrImageLoader.java:183)
	at org.embl.mobie.io.ome.zarr.loaders.N5OMEZarrImageLoader.fetchSequenceDescriptionAndViewRegistrations(N5OMEZarrImageLoader.java:134)
	at org.embl.mobie.io.ome.zarr.loaders.N5OMEZarrImageLoader.<init>(N5OMEZarrImageLoader.java:122)
	at org.embl.mobie.io.ome.zarr.openers.OMEZarrOpener.readFile(OMEZarrOpener.java:45)
	at org.embl.mobie.io.ome.zarr.openers.OMEZarrOpener.openFile(OMEZarrOpener.java:23)
	at org.embl.mobie.viewer.command.OpenOMEZARRCommand.openAndShow(OpenOMEZARRCommand.java:20)
	at org.embl.mobie.viewer.command.OpenOMEZARRCommand.run(OpenOMEZARRCommand.java:28)
	at org.scijava.command.CommandModule.run(CommandModule.java:196)

The text was updated successfully, but these errors were encountered:

constantinpape · 2022-02-17T12:55:01Z

It's an encoding issue: the data is written with $\um$m as unit (which is also discouraged by the spec). If you change it to some unit name that is UTF-8 it works. (This is why sticking to UTF-8 is always a good idea....)

tischi · 2022-02-17T13:00:41Z

This file was written with bioformats2raw (ping @sbesson @joshmoore), which is what many people use to create OME-Zarr files.
What should we do now?
Can we support this encoding in mobie-io?
Should bioformats2raw be changed?

constantinpape · 2022-02-17T13:16:02Z

What should we do now?
Can we support this encoding in mobie-io?
Should bioformats2raw be changed?

Ideally this should be fixed in bioformats to raw, so that it writes utf-8 and only uses the recommended names for units in order to be fully spec compliant.

sbesson · 2022-02-17T13:33:41Z

Can you share the extract of the JSON which causes the issue on Mobie?
If it is possible to create a minimal reproducible scenario, this is likely something to report as an issue on https://github.com/glencoesoftware/bioformats2raw

constantinpape · 2022-02-17T13:58:16Z

Can you share the extract of the JSON which causes the issue on Mobie?
If it is possible to create a minimal reproducible scenario, this is likely something to report as an issue on https://github.com/glencoesoftware/bioformats2raw

@tischi can you provide the minimal example and data for this?

I think we can go ahead with the release irrespective of this.

bugraoezdemir · 2022-02-17T15:18:24Z

Here is the example data:
https://oc.embl.de/index.php/s/keVoyhMXY2IqvJF

Here is the associated information including the conversion parameters:
minimal_example.docx

constantinpape · 2022-02-17T15:27:48Z

@bugraoezdemir
This is not the same metadata that causes the issue:

{
  "multiscales" : [
    {
      "metadata" : {
        "method" : "loci.common.image.SimpleImageScaler",
        "version" : "Bio-Formats 6.8.0"
      },
      "datasets" : [
        {
          "path" : "0"
        },
        {
          "path" : "1"
        },
        {
          "path" : "2"
        }
      ],
      "version" : "0.2"
    }
  ]
}

I will try to discuss this with @tischi I am a bit confused here.

tischi · 2022-02-17T15:54:44Z

(Fiji Is Just) ImageJ 2.3.0/1.53f51; Java 1.8.0_202 [64-bit]; Mac OS X 10.14.6; 67MB of 9749MB (<1%)
 
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
	at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:307)
	at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166)
	at ij.IJ.runPlugIn(IJ.java)
	at ij.Executer.runCommand(Executer.java:150)
	at ij.Executer.run(Executer.java:68)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Module threw exception
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:303)
	... 5 more
Caused by: java.lang.RuntimeException: Module threw exception
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
Caused by: java.lang.NullPointerException
	at org.embl.mobie.io.ome.zarr.util.ZArrayAttributes.getDatasetAttributes(ZArrayAttributes.java:69)
	at org.embl.mobie.io.ome.zarr.readers.N5OmeZarrReader.getDatasetAttributes(N5OmeZarrReader.java:232)
	at org.embl.mobie.io.ome.zarr.loaders.N5OMEZarrImageLoader.getDatasetAttributes(N5OMEZarrImageLoader.java:248)
	at org.embl.mobie.io.ome.zarr.loaders.N5OMEZarrImageLoader.initSetups(N5OMEZarrImageLoader.java:185)
	at org.embl.mobie.io.ome.zarr.loaders.N5OMEZarrImageLoader.fetchSequenceDescriptionAndViewRegistrations(N5OMEZarrImageLoader.java:134)
	at org.embl.mobie.io.ome.zarr.loaders.N5OMEZarrImageLoader.<init>(N5OMEZarrImageLoader.java:122)
	at org.embl.mobie.io.ome.zarr.openers.OMEZarrOpener.readFile(OMEZarrOpener.java:45)
	at org.embl.mobie.io.ome.zarr.openers.OMEZarrOpener.openFile(OMEZarrOpener.java:23)
	at org.embl.mobie.viewer.command.OpenOMEZARRCommand.openAndShow(OpenOMEZARRCommand.java:20)
	at org.embl.mobie.viewer.command.OpenOMEZARRCommand.run(OpenOMEZARRCommand.java:28)

This is the error that corresponds to the image @bugraoezdemir uploaded.

bugraoezdemir · 2022-02-17T15:58:27Z

Sorry. It was my mistake. The earlier error corresponded to another conversion.

constantinpape · 2022-02-17T16:57:19Z

Hey @sbesson,
we had a closer look at this now and I initially mixed up some files and mistook the error.
So the issue here is that by default bioformats2raw writes a nested structure that we can't deal with in MoBIE.
But there is an option to write to the root level directly, which then should work for us.
So for now it seems like everything is ok in bioformats2raw; sorry for the false alarm.
(As a sidenote, we still have issues to open the no hierarchy version of the file, see #73; but this might be due to the fact that this is ome.zarr v0.2, which we haven't tested much in MoBIE.)

tischi · 2022-02-17T17:38:17Z

writes a nested structure that we can't deal with in MoBIE.

Now I am a bit confused 😄
I thought we can open a zarr with multiple images in MoBIE?
Doesn't that imply a nested folder structure, i.e. no multi-scales at the top level?

constantinpape · 2022-02-17T17:45:43Z

I thought we can open a zarr with multiple images in MoBIE?
Doesn't that imply a nested folder structure, i.e. no multi-scales at the top level?

Yes, we can open multiple images, but that does not imply a nested folder structure w.r.t. the multi-scales metadata:

We can open (sorry for the filetree / json hybrid):

image.ome.zarr/
  .zattrs: {"multiscales": [{"name": "image1", ...}, {"name": "image2", ...}]}

but we can't open:

image.ome.zarr/
  image1/.zattrs: {"multiscales": [{"name": "image1", ....}]}
  image2/.zattrs: {"multiscales": [{"name": "image2", ...}]}

tischi · 2022-02-17T17:56:29Z

Oh, wow, I did not even know that the spec allowed multiple images within one multiscale (I will refrain from asking what's the use-case for that in order not to abuse this issues (Praeteritio)).

constantinpape · 2022-02-17T19:32:48Z

@tischi your past self has wondered about this already: ome/ngff#17
;)

sbesson · 2022-02-17T22:23:12Z

I will leave you guys to sort out what MoBIE should or should not support 😄 Cross-linking other threads for completion:

the default hierarchy produced by bioformats2raw 0.4.0 is indeed a top-level Zarr group containing one or more multiscale OME-NGFF images as well as an OME group with the OME-XML metadata (see also Documentation & parsing ome-zarr with original zarr package ome/ome-zarr-py#163 (comment)).
a converter option exists which allows to flatten this layout and only generate the multiscales image - see https://forum.image.sc/t/dropping-the-top-layer-in-ome-zarr-data-without-losing-multiscales-metadata/63286
bioformats2raw 0.4.0 writes metadata following the 0.2 version of the OME-NGFF specification but work is currently ongoing to upgrade the implementation to comply with the recently released 0.4 version of the specification - see Store dimension order using new "axes" spec glencoesoftware/bioformats2raw#121. I expect this should maximize the compatibility with MoBIE

tischi · 2022-02-18T07:46:54Z

@constantinpape
Thanks for pointing out my past self.
Then I don't really think we can speak of being able to open multiple images in MoBIE.
This is more several times the same image, at potentially different chunking....

constantinpape · 2022-02-18T08:52:02Z

@sbesson thanks for the pointers!

the default hierarchy produced by bioformats2raw 0.4.0 is indeed a top-level Zarr group containing one or more multiscale OME-NGFF images as well as an OME group with the OME-XML metadata (see also

a converter option exists which allows to flatten this layout and only generate the multiscales image - see https://forum.image.sc/t/dropping-the-top-layer-in-ome-zarr-data-without-losing-multiscales-metadata/63286

Yes, we (at least the @bugraoezdemir and @tischi who are actually using bioformats2raw) are aware of the converter option now and will use it to create ome.zarr containers compatible with MoBIE.
We also want to support the nested hierarchies eventually, but it makes most sense to do this once we integrate with the metadata functionality from https://github.com/saalfeldlab/n5-imglib2, which has support for nesting already.

And thanks for the xref to glencoesoftware/bioformats2raw#121. Do you have a rough ETA on when that will be merged and released?

Then I don't really think we can speak of being able to open multiple images in MoBIE.

Yes, we don't support it in the very general case of a nested hierarchy yet.

This is more several times the same image, at potentially different chunking....

There is no restriction on how to use multiple images at the same hierarchy level, so it might as well be different images.

constantinpape · 2022-03-15T17:34:09Z

This has been resolved.

tischi added the bug Something isn't working label Feb 17, 2022

constantinpape closed this as completed Mar 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error when opening OME-Zarr created with bioformats2raw #72

Error when opening OME-Zarr created with bioformats2raw #72

tischi commented Feb 17, 2022

constantinpape commented Feb 17, 2022

tischi commented Feb 17, 2022

constantinpape commented Feb 17, 2022 •

edited

Loading

sbesson commented Feb 17, 2022

constantinpape commented Feb 17, 2022

bugraoezdemir commented Feb 17, 2022

constantinpape commented Feb 17, 2022

tischi commented Feb 17, 2022

bugraoezdemir commented Feb 17, 2022

constantinpape commented Feb 17, 2022

tischi commented Feb 17, 2022

constantinpape commented Feb 17, 2022 •

edited

Loading

tischi commented Feb 17, 2022

constantinpape commented Feb 17, 2022

sbesson commented Feb 17, 2022

tischi commented Feb 18, 2022

constantinpape commented Feb 18, 2022 •

edited

Loading

constantinpape commented Mar 15, 2022

Error when opening OME-Zarr created with bioformats2raw #72

Error when opening OME-Zarr created with bioformats2raw #72

Comments

tischi commented Feb 17, 2022

constantinpape commented Feb 17, 2022

tischi commented Feb 17, 2022

constantinpape commented Feb 17, 2022 • edited Loading

sbesson commented Feb 17, 2022

constantinpape commented Feb 17, 2022

bugraoezdemir commented Feb 17, 2022

constantinpape commented Feb 17, 2022

tischi commented Feb 17, 2022

bugraoezdemir commented Feb 17, 2022

constantinpape commented Feb 17, 2022

tischi commented Feb 17, 2022

constantinpape commented Feb 17, 2022 • edited Loading

tischi commented Feb 17, 2022

constantinpape commented Feb 17, 2022

sbesson commented Feb 17, 2022

tischi commented Feb 18, 2022

constantinpape commented Feb 18, 2022 • edited Loading

constantinpape commented Mar 15, 2022

constantinpape commented Feb 17, 2022 •

edited

Loading

constantinpape commented Feb 17, 2022 •

edited

Loading

constantinpape commented Feb 18, 2022 •

edited

Loading