Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZarrReader: HCS validation fixes #24

Merged
merged 7 commits into from
Apr 28, 2022
Merged

Conversation

dgault
Copy link
Member

@dgault dgault commented Mar 29, 2022

This includes a number of fixes for validation issues reported in #22
It also includes some additional fixes for multi res HCS data discovered during testing.

To test you can run the below and verify that no validation issues are reported and that the resolution levels are properly detected:
BF_CP=/bio-formats-build/ZarrReader/target/OMEZarrReader-0.1.3-SNAPSHOT-jar-with-dependencies.jar /bio-formats-build/bioformats/tools/showinf -nopix -noflat -omexml /uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/.zattrs

Fixes #22

@snoopycrimecop
Copy link
Member

snoopycrimecop commented Mar 30, 2022

Conflicting PR. Removed from build BIOFORMATS-push#51. See the console output for more details.
Possible conflicts:

--conflicts Conflict resolved in build BIOFORMATS-push#61. See the console output for more details.

@snoopycrimecop
Copy link
Member

snoopycrimecop commented Apr 20, 2022

Conflicting PR. Removed from build BIOFORMATS-push#72. See the console output for more details.
Possible conflicts:

  • Upstream changes
    • src/loci/formats/in/ZarrReader.java

--conflicts Conflict resolved in build BIOFORMATS-push#1211. See the console output for more details.

@sbesson
Copy link
Member

sbesson commented Apr 20, 2022

It looks like origin/main will need to be merge into this branch now that #29 has been included

@joshmoore
Copy link
Member

@dgault: I've pushed a merge commit if wouldn't mind reviewing it.

@snoopycrimecop snoopycrimecop mentioned this pull request Apr 21, 2022
@dgault
Copy link
Member Author

dgault commented Apr 21, 2022

There are 4 main things to test with this PR:

Validation fixes: This relates to the original issue reported: #22
Testing using idr/idr0094A/7751.zarr and showinf -nopix -omexml should no longer report any validation warnings

8 bit data: Without this PR 8 bit data would display as a blank plane. With the PR included 8bit data should display correctly. Any sample file with 8bit pixel data should be suitable for testing, Petr's data at uod/idr-scratch/test.zarr should be suitable.

Multi Image, Multi Resolutions: Without this PR multi resolutions on the first image were detected correctly but for subsequent images the resolution levels were not detected. With this PR the resolution levels should be correctly detected for all images. This can be tested using v0.4/multi-image.ome.zarr

Default DimensionOrder: This was discovered using Petr's test data, in this case the dimension order is not present in the Zarr attributes and the default value was being used, however without this PR there was an inconsistency in the reader for this default order. This has now been corrected to use XYZCT. The file to test would be uod/idr-scratch/petr-test/multifile-Z1.ome.zarr. This file should have 5 z slices, without this PR it would display as 5 timepoints or just as a single plane (the XML would be correct and could lead to some odd behaviour in the client).

@sbesson sbesson requested review from sbesson and pwalczysko April 21, 2022 15:40
@pwalczysko
Copy link
Member

pwalczysko commented Apr 22, 2022

Testing

8 bit data issue (nr. 2 from #24 (comment))
together with
Default Dimension Order issue (nr. 4 from #24 (comment))
Test with this PR in Fiji
Pixel data look good
Screenshot 2022-04-22 at 16 30 59

Dimension order looks good
Screenshot 2022-04-22 at 16 27 55

The image is z=5 as expected.

Without this PR
Pixel data are blank on z 2-5. But the other problems detected in OMERO are not visible.

This looks good, but maybe for other tests we would profit from having merge-ci back ? Or any suggestions for usage of an OMERO server ?

@pwalczysko
Copy link
Member

pwalczysko commented Apr 22, 2022

Tried to test also Ad 3 Multi-image multi resolutions from #24 (comment).
Without this PR I am getting locally (@dgault advised that the errors in the output below are rather connected with dimenstion order, no validation errors are visible)

BF_CP=/Users/pwalczysko/Downloads/OMEZarrReader-0.1.5-withoutPR-jar-with-dependencies.jar /Users/pwalczysko/Downloads/bftools/showinf -nopix -noflat -omexml  /Users/pwalczysko/ngff-testing/multi-image.ome.zarr/.zattrs
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/pwalczysko/Downloads/bftools/bioformats_package.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/pwalczysko/Downloads/OMEZarrReader-0.1.5-withoutPR-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
Checking file format [Zarr]
Initializing reader
ZarrReader initializing /Users/pwalczysko/ngff-testing/multi-image.ome.zarr/.zattrs
Invalid dimension order: XYXYXYXYCTZ
ome.xml.model.enums.EnumerationException: 'XYXYXYXYCTZ' not a supported value of 'class ome.xml.model.enums.DimensionOrder'
	at ome.xml.model.enums.DimensionOrder.fromString(DimensionOrder.java:106)
	at loci.formats.MetadataTools.populatePixelsOnly(MetadataTools.java:299)
	at loci.formats.MetadataTools.populateMetadata(MetadataTools.java:262)
	at loci.formats.MetadataTools.populatePixels(MetadataTools.java:151)
	at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:354)
	at loci.formats.FormatReader.setId(FormatReader.java:1443)
	at loci.formats.ImageReader.setId(ImageReader.java:849)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1035)
	at loci.formats.tools.ImageInfo.main(ImageInfo.java:1121)
Exception in thread "main" java.lang.IllegalArgumentException: Invalid dimension order: XYXYXYXYCTZ
	at loci.formats.FormatTools.getZCTCoords(FormatTools.java:615)
	at loci.formats.FormatTools.getZCTCoords(FormatTools.java:565)
	at loci.formats.FormatReader.getZCTCoords(FormatReader.java:1140)
	at loci.formats.MetadataTools.populatePixels(MetadataTools.java:196)
	at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:354)
	at loci.formats.FormatReader.setId(FormatReader.java:1443)
	at loci.formats.ImageReader.setId(ImageReader.java:849)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1035)
	at loci.formats.tools.ImageInfo.main(ImageInfo.java:1121)

With this PR I am getting no errors . Looks good. (edited)

BF_CP=/Users/pwalczysko/Downloads/OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar /Users/pwalczysko/Downloads/bftools/showinf -nopix -noflat -omexml  /Users/pwalczysko/ngff-testing/multi-image.ome.zarr/.zattrs
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/pwalczysko/Downloads/bftools/bioformats_package.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/pwalczysko/Downloads/OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
Checking file format [Zarr]
Initializing reader
ZarrReader initializing /Users/pwalczysko/ngff-testing/multi-image.ome.zarr/.zattrs
Parsing schema path
http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd
Validating OME-XML
No validation errors found.
Initialization took 0.709s

Reading core metadata
filename = /Users/pwalczysko/ngff-testing/multi-image.ome.zarr/.zattrs
Used files:
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/.zattrs
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/.zgroup
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s2/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s2/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/.zattrs
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/.zgroup
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s1/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s1/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s1/0/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s1/1/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s1/1/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/0/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/0/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/0/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/1/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/1/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/1/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/1/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/3/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/3/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/3/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/3/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/2/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/2/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/2/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-3/s0/2/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s2/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s2/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/.zattrs
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/.zgroup
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s1/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s1/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s1/0/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s1/1/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s1/1/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/0/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/0/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/0/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/1/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/1/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/1/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/1/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/3/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/3/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/3/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/3/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/2/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/2/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/2/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-2/s0/2/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s2/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s2/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/.zattrs
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/.zgroup
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s1/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s1/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s1/0/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s1/1/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s1/1/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/0/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/0/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/0/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/1/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/1/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/1/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/1/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/3/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/3/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/3/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/3/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/2/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/2/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/2/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-0/s0/2/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s2/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s2/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/.zattrs
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/.zgroup
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s1/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s1/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s1/0/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s1/1/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s1/1/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/.zarray
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/0/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/0/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/0/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/0/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/1/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/1/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/1/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/1/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/3/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/3/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/3/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/3/2
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/2/0
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/2/1
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/2/3
	/Users/pwalczysko/ngff-testing/multi-image.ome.zarr/image-1/s0/2/2
Series count = 4
Series #0 -- image-0/s0:
	Resolutions = 3
		sizeX[0] = 1024
		sizeX[1] = 512
		sizeX[2] = 256
	Image count = 1
	RGB = false (1) 
	Interleaved = false
	Indexed = false (false color)
	Width = 1024
	Height = 930
	SizeZ = 1
	SizeT = 1
	SizeC = 1
	Tile size = 256 x 256
	Thumbnail size = 128 x 116
	Endianness = intel (little)
	Dimension order = XYCTZ (uncertain)
	Pixel type = uint16
	Valid bits per pixel = 16
	Metadata complete = false
	Thumbnail series = false
	-----
	Plane #0 <=> Z 0, C 0, T 0

Series #1 -- image-1/s0:
	Resolutions = 3
		sizeX[0] = 1024
		sizeX[1] = 512
		sizeX[2] = 256
	Image count = 1
	RGB = false (1) 
	Interleaved = false
	Indexed = false (false color)
	Width = 1024
	Height = 930
	SizeZ = 1
	SizeT = 1
	SizeC = 1
	Tile size = 256 x 256
	Thumbnail size = 128 x 116
	Endianness = intel (little)
	Dimension order = XYCTZ (uncertain)
	Pixel type = uint16
	Valid bits per pixel = 16
	Metadata complete = false
	Thumbnail series = false
	-----
	Plane #0 <=> Z 0, C 0, T 0

Series #2 -- image-2/s0:
	Resolutions = 3
		sizeX[0] = 1024
		sizeX[1] = 512
		sizeX[2] = 256
	Image count = 1
	RGB = false (1) 
	Interleaved = false
	Indexed = false (false color)
	Width = 1024
	Height = 930
	SizeZ = 1
	SizeT = 1
	SizeC = 1
	Tile size = 256 x 256
	Thumbnail size = 128 x 116
	Endianness = intel (little)
	Dimension order = XYCTZ (uncertain)
	Pixel type = uint16
	Valid bits per pixel = 16
	Metadata complete = false
	Thumbnail series = false
	-----
	Plane #0 <=> Z 0, C 0, T 0

Series #3 -- image-3/s0:
	Resolutions = 3
		sizeX[0] = 1024
		sizeX[1] = 512
		sizeX[2] = 256
	Image count = 1
	RGB = false (1) 
	Interleaved = false
	Indexed = false (false color)
	Width = 1024
	Height = 930
	SizeZ = 1
	SizeT = 1
	SizeC = 1
	Tile size = 256 x 256
	Thumbnail size = 128 x 116
	Endianness = intel (little)
	Dimension order = XYCTZ (uncertain)
	Pixel type = uint16
	Valid bits per pixel = 16
	Metadata complete = false
	Thumbnail series = false
	-----
	Plane #0 <=> Z 0, C 0, T 0


Reading global metadata
Axis name:0:0: y
Axis name:0:1: x
Axis name:1:0: y
Axis name:1:1: x
Axis name:2:0: y
Axis name:2:1: x
Axis name:3:0: y
Axis name:3:1: x
Axis type:0:0: space
Axis type:0:1: space
Axis type:1:0: space
Axis type:1:1: space
Axis type:2:0: space
Axis type:2:1: space
Axis type:3:0: space
Axis type:3:1: space

Reading series #0 metadata

Generating OME-XML (schema version 2016-06)
<?xml version="1.0" encoding="UTF-8"?>
<OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2016-06" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2016-06 http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd">
   <Image ID="Image:0" Name="image-0/s0">
      <Pixels BigEndian="false" DimensionOrder="XYCTZ" ID="Pixels:0" Interleaved="false" SignificantBits="16" SizeC="1" SizeT="1" SizeX="1024" SizeY="930" SizeZ="1" Type="uint16">
         <Channel ID="Channel:0:0" SamplesPerPixel="1">
            <LightPath/>
         </Channel>
         <MetadataOnly/>
         <Plane TheC="0" TheT="0" TheZ="0"/>
      </Pixels>
   </Image>
   <Image ID="Image:1" Name="image-1/s0">
      <Pixels BigEndian="false" DimensionOrder="XYCTZ" ID="Pixels:1" Interleaved="false" SignificantBits="16" SizeC="1" SizeT="1" SizeX="1024" SizeY="930" SizeZ="1" Type="uint16">
         <Channel ID="Channel:1:0" SamplesPerPixel="1">
            <LightPath/>
         </Channel>
         <MetadataOnly/>
         <Plane TheC="0" TheT="0" TheZ="0"/>
      </Pixels>
   </Image>
   <Image ID="Image:2" Name="image-2/s0">
      <Pixels BigEndian="false" DimensionOrder="XYCTZ" ID="Pixels:2" Interleaved="false" SignificantBits="16" SizeC="1" SizeT="1" SizeX="1024" SizeY="930" SizeZ="1" Type="uint16">
         <Channel ID="Channel:2:0" SamplesPerPixel="1">
            <LightPath/>
         </Channel>
         <MetadataOnly/>
         <Plane TheC="0" TheT="0" TheZ="0"/>
      </Pixels>
   </Image>
   <Image ID="Image:3" Name="image-3/s0">
      <Pixels BigEndian="false" DimensionOrder="XYCTZ" ID="Pixels:3" Interleaved="false" SignificantBits="16" SizeC="1" SizeT="1" SizeX="1024" SizeY="930" SizeZ="1" Type="uint16">
         <Channel ID="Channel:3:0" SamplesPerPixel="1">
            <LightPath/>
         </Channel>
         <MetadataOnly/>
         <Plane TheC="0" TheT="0" TheZ="0"/>
      </Pixels>
   </Image>
   <StructuredAnnotations>
      <XMLAnnotation ID="Annotation:0">
         <Value>{
  "multiscales" : [ {
    "axes" : [ {
      "name" : "y",
      "type" : "space"
    }, {
      "name" : "x",
      "type" : "space"
    } ],
    "datasets" : [ {
      "coordinateTransformations" : [ {
        "scale" : [ 1.0, 1.0 ],
        "type" : "scale"
      } ],
      "path" : "image-0/s0"
    }, {
      "coordinateTransformations" : [ {
        "scale" : [ 2.0, 2.0 ],
        "type" : "scale"
      } ],
      "path" : "image-0/s1"
    }, {
      "coordinateTransformations" : [ {
        "scale" : [ 4.0, 4.0 ],
        "type" : "scale"
      } ],
      "path" : "image-0/s2"
    } ],
    "metadata" : {
      "order" : 0,
      "preserve_range" : true,
      "scale" : [ 0.5, 0.5 ]
    },
    "name" : "image-0",
    "type" : "skimage.transform._warps.rescale",
    "version" : "0.4"
  }, {
    "axes" : [ {
      "name" : "y",
      "type" : "space"
    }, {
      "name" : "x",
      "type" : "space"
    } ],
    "datasets" : [ {
      "coordinateTransformations" : [ {
        "scale" : [ 1.0, 1.0 ],
        "type" : "scale"
      } ],
      "path" : "image-1/s0"
    }, {
      "coordinateTransformations" : [ {
        "scale" : [ 2.0, 2.0 ],
        "type" : "scale"
      } ],
      "path" : "image-1/s1"
    }, {
      "coordinateTransformations" : [ {
        "scale" : [ 4.0, 4.0 ],
        "type" : "scale"
      } ],
      "path" : "image-1/s2"
    } ],
    "metadata" : {
      "order" : 0,
      "preserve_range" : true,
      "scale" : [ 0.5, 0.5 ]
    },
    "name" : "image-1",
    "type" : "skimage.transform._warps.rescale",
    "version" : "0.4"
  }, {
    "axes" : [ {
      "name" : "y",
      "type" : "space"
    }, {
      "name" : "x",
      "type" : "space"
    } ],
    "datasets" : [ {
      "coordinateTransformations" : [ {
        "scale" : [ 1.0, 1.0 ],
        "type" : "scale"
      } ],
      "path" : "image-2/s0"
    }, {
      "coordinateTransformations" : [ {
        "scale" : [ 2.0, 2.0 ],
        "type" : "scale"
      } ],
      "path" : "image-2/s1"
    }, {
      "coordinateTransformations" : [ {
        "scale" : [ 4.0, 4.0 ],
        "type" : "scale"
      } ],
      "path" : "image-2/s2"
    } ],
    "metadata" : {
      "order" : 0,
      "preserve_range" : true,
      "scale" : [ 0.5, 0.5 ]
    },
    "name" : "image-2",
    "type" : "skimage.transform._warps.rescale",
    "version" : "0.4"
  }, {
    "axes" : [ {
      "name" : "y",
      "type" : "space"
    }, {
      "name" : "x",
      "type" : "space"
    } ],
    "datasets" : [ {
      "coordinateTransformations" : [ {
        "scale" : [ 1.0, 1.0 ],
        "type" : "scale"
      } ],
      "path" : "image-3/s0"
    }, {
      "coordinateTransformations" : [ {
        "scale" : [ 2.0, 2.0 ],
        "type" : "scale"
      } ],
      "path" : "image-3/s1"
    }, {
      "coordinateTransformations" : [ {
        "scale" : [ 4.0, 4.0 ],
        "type" : "scale"
      } ],
      "path" : "image-3/s2"
    } ],
    "metadata" : {
      "order" : 0,
      "preserve_range" : true,
      "scale" : [ 0.5, 0.5 ]
    },
    "name" : "image-3",
    "type" : "skimage.transform._warps.rescale",
    "version" : "0.4"
  } ]
}</Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:1">
         <Value>{
  "_ARRAY_DIMENSIONS" : [ "y", "x" ]
}</Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:2">
         <Value>{
  "_ARRAY_DIMENSIONS" : [ "y", "x" ]
}</Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:3">
         <Value>{
  "_ARRAY_DIMENSIONS" : [ "y", "x" ]
}</Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:4">
         <Value>{
  "_ARRAY_DIMENSIONS" : [ "y", "x" ]
}</Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:5" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis name:3:1</Key>
               <Value>x</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:6" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis name:3:0</Key>
               <Value>y</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:7" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis name:2:1</Key>
               <Value>x</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:8" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis name:2:0</Key>
               <Value>y</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:9" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis name:1:1</Key>
               <Value>x</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:10" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis name:1:0</Key>
               <Value>y</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:11" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis name:0:1</Key>
               <Value>x</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:12" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis name:0:0</Key>
               <Value>y</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:13" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis type:3:1</Key>
               <Value>space</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:14" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis type:3:0</Key>
               <Value>space</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:15" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis type:2:1</Key>
               <Value>space</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:16" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis type:2:0</Key>
               <Value>space</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:17" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis type:1:1</Key>
               <Value>space</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:18" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis type:0:1</Key>
               <Value>space</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:19" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis type:1:0</Key>
               <Value>space</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:20" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis type:0:0</Key>
               <Value>space</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
   </StructuredAnnotations>
</OME>

@pwalczysko
Copy link
Member

@sbesson happy to discuss the outputs in #24 (comment) in more detail, - is viewing of the image necessary ?

@pwalczysko
Copy link
Member

pwalczysko commented Apr 24, 2022

Testing the 8 bit data paragraph on OMERO did fully succeed.

Workflow:

  1. Get the prebuilt OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar with this PR from @dgault and rsync it onto idr0-slot3:/tmp.
  2. [pwalczysko@idr0-slot3 ~]$ sudo cp /tmp/OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar /uod/idr/scratch/idr2-slot3/petr-test/
  3. Then ssh into idr2-slot3 and start the docker where merge-ci is as per the https://hackmd.io/hmVE6CmATROmrAim39Uo5w
  4. (.venv3) bash-4.2$ cp /uod/idr-scratch/petr-test/OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar ./uod/idr-scratch/petr-test/OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar
  5. then copy the OMEZarr...jar into the lib/server and lib/client as per the https://hackmd.io/hmVE6CmATROmrAim39Uo5w, this means
(.venv3) bash-4.2$ pwd
/home/omero/workspace/OMERO-server/OMERO.server/lib/server
(.venv3) bash-4.2$ cp /uod/idr-scratch/petr-test/OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar 
...
(.venv3) bash-4.2$ pwd
/home/omero/workspace/OMERO-server/OMERO.server/lib/client
(.venv3) bash-4.2$ cp /uod/idr-scratch/petr-test/OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar .
  1. import the file in-place, all goes smoothly
  2. omero import --transfer=ln_s --skip=checksum --parallel-upload=10 --depth 10 /uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/

Also see import -f output

omero import -f  /uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/omero/workspace/OMERO-server/OMERO.server/lib/client/logback-classic.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/omero/workspace/OMERO-server/OMERO.server/lib/client/OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2022-04-24 12:14:23,805 220        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.11-SNAPSHOT
2022-04-24 12:14:23,822 237        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.10.0-SNAPSHOT revision: fe08d544114e0f95e194a6c67a0e09c952864096 date: 24 April 2022
2022-04-24 12:14:23,866 281        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2022-04-24 12:14:24,218 633        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2022-04-24 12:14:24,799 1214       [      main] INFO      ome.formats.importer.ImportCandidates - 6 file(s) parsed into 1 group(s) with 1 call(s) to setId in 571ms. (581ms total) [0 unknowns]
#======================================
# Group: /uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/OME/METADATA.ome.xml SPW: false Reader: loci.formats.in.ZarrReader
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/OME/METADATA.ome.xml
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/.zattrs
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/.zattrs
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/.zarray
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/0/0/3/0/0
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/0/0/2/0/0
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/0/0/4/0/0
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/0/0/1/0/0
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/0/0/0/0/0
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/.zgroup
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/.zgroup

The positive the dimensions of the image (t=1, z=5) and the number of the images (just 1) is correct.

Edited: after OMERO.server restart and reimport, the image has the correct pixel values. See user-3 https://merge-ci.openmicroscopy.org/web/webclient/?show=image-229601

@pwalczysko
Copy link
Member

@dgault @sbesson edited #24 (comment) - after server restart and reimport, all looks good.

@pwalczysko
Copy link
Member

Re: Validation fixes (point 1 of #24 (comment)

BF_CP=OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar  bftools/showinf -nopix -omexml /uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/.zattrs
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/uod/idr-scratch/petr-test/bftools/bioformats_package.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/uod/idr-scratch/petr-test/OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
Checking file format [Zarr]
Initializing reader
ZarrReader initializing /uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/.zattrs
Exception in thread "main" java.lang.IllegalArgumentException: Invalid index: -1
	at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328)
	at loci.formats.in.ZarrReader.parseWells(ZarrReader.java:686)
	at loci.formats.in.ZarrReader.parsePlate(ZarrReader.java:656)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:362)
	at loci.formats.FormatReader.setId(FormatReader.java:1443)
	at loci.formats.ImageReader.setId(ImageReader.java:849)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1035)
	at loci.formats.tools.ImageInfo.main(ImageInfo.java:1121)

@pwalczysko
Copy link
Member

In summary, the tests in #24 (comment) are as follows:

Validation fixes fails for me with #24 (comment) (tested on idr2-slot3)
8-bit.. fully passes on Fiji and OMERO
Multi Image, Multi Resolutions passes on the CLI using BF tools, tested locally MacOS
Default DimensionOrder: passes fully on Fiji and OMERO

Seems to me out of 4 tests, 2 full passes, 1 pass with question (OMERO test needed ?) and 1 fail.

Copy link
Member

@pwalczysko pwalczysko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please comment on the fail of the Validation fixes test

@dgault
Copy link
Member Author

dgault commented Apr 26, 2022

It seems something has likely broken during the conflict merges or later fixes. I will try and get another commit to resolve the issue and relist for testing when ready

@pwalczysko
Copy link
Member

The error reported in #24 (comment) is now fixed

BF_CP=OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar  bftools/showinf -nopix -omexml /uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/.zattrs
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/uod/idr-scratch/petr-test/bftools/bioformats_package.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/uod/idr-scratch/petr-test/OMEZarrReader-0.1.5-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
Checking file format [Zarr]
Initializing reader
ZarrReader initializing /uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/.zattrs
Parsing schema path
http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd
Validating OME-XML
No validation errors found.
Initialization took 53.179s

Reading core metadata
filename = /uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/.zattrs
Used files:
......
182000 of lines of output
....
}</Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:961" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis:0:1</Key>
               <Value>x</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
      <XMLAnnotation ID="Annotation:962" Namespace="openmicroscopy.org/OriginalMetadata">
         <Value>
            <OriginalMetadata>
               <Key>Axis:0:0</Key>
               <Value>y</Value>
            </OriginalMetadata>
         </Value>
      </XMLAnnotation>
   </StructuredAnnotations>
</OME>

@pwalczysko
Copy link
Member

In summary, all looks good. The error is fixed, all files are having either a good omero import -f output or were imported to omero.
The only slight worry is that import to omero of
/uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/ somehow expires without warning after a long wait. But import -f is looking sound.

omero import -f  /uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/
2022-04-28 13:43:52,046 253        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.11-SNAPSHOT
2022-04-28 13:43:52,073 280        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.10.0-SNAPSHOT revision: a3e43d1b3c1c1cbbae7d42a4265d67c1c0c3b594 date: 28 April 2022
2022-04-28 13:43:52,128 335        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2022-04-28 13:43:52,488 695        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2022-04-28 13:43:54,530 2737       [      main] INFO      ome.formats.importer.ImportCandidates - 1930 file(s) parsed into 0 group(s) with 0 call(s) to setId in 0ms. (2042ms total) [0 unknowns]
(.venv3) bash-4.2$ omero import -f  --depth 10 /uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/
2022-04-28 13:45:16,487 214        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.11-SNAPSHOT
2022-04-28 13:45:16,504 231        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.10.0-SNAPSHOT revision: a3e43d1b3c1c1cbbae7d42a4265d67c1c0c3b594 date: 28 April 2022
2022-04-28 13:45:16,546 273        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2022-04-28 13:45:16,899 626        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2022-04-28 13:45:48,264 31991      [      main] INFO      ome.formats.importer.ImportCandidates - 10570 file(s) parsed into 1 group(s) with 1 call(s) to setId in 22029ms. (31364ms total) [0 unknowns]
#======================================
# Group: /uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/G/7/7/3/0/0 SPW: false Reader: loci.formats.in.ZarrReader
/uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/G/7/7/3/0/0
/uod/idr/repos/curated/unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr/G/7/7/3/.zarray
....

@sbesson sbesson merged commit 33deebc into ome:main Apr 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

HCS validation errors
5 participants