Read multi-resolution pyramids in Well class (closes #208) #209

tcompa · 2022-06-27T09:25:14Z

Fixes #208, by replicating a structure similar to the Plate class into the Well class (i.e. reading multi-resolution pyramids).

Logs for napari -vvv --plugin napari-ome-zarr xxx.zarr/B/03/ (also including a bit of zooming in/out, to trigger more loading) show that different levels are actually loaded:

Click to expand!

11:20:51 DEBUG Created nested FSStore(xxx.zarr/B/03/, r, {'dimension_separator': '/', 'normalize_keys': False})
11:20:51 DEBUG 0.4 matches 0.3?
11:20:51 DEBUG 0.3 matches 0.3?
11:20:51 DEBUG ZarrLocation.__init__ xxx.zarr/B/03/ detected:FormatV03
11:20:51 WARNING version mismatch: detected:FormatV03, requested:FormatV04
11:20:51 DEBUG Created nested FSStore(xxx.zarr/B/03/, r, {'dimension_separator': '/', 'normalize_keys': False})
11:20:51 DEBUG treating /tmp/xxx.zarr/B/03 [zgroup] as Well
11:20:51 INFO root_attr: well
11:20:51 DEBUG {'images': [{'path': '0'}], 'version': '0.3'}
11:20:51 INFO well_data: {'images': [{'path': '0'}], 'version': '0.3'}
11:20:51 DEBUG open(ZarrLocation(/tmp/xxx.zarr/B/03/0))
11:20:51 DEBUG ZarrLocation.__init__ path:/tmp/xxx.zarr/B/03/0, fmt:0.3
11:20:51 DEBUG Created nested FSStore(/tmp/xxx.zarr/B/03/0, r, {'dimension_separator': '/', 'normalize_keys': False})
11:20:51 DEBUG 0.4 matches 0.3?
11:20:51 DEBUG 0.3 matches 0.3?
11:20:51 DEBUG ZarrLocation.__init__ /tmp/xxx.zarr/B/03/0 detected:FormatV03
11:20:51 DEBUG treating /tmp/xxx.zarr/B/03/0 [zgroup] as Multiscales
11:20:51 INFO root_attr: multiscales
11:20:51 DEBUG [{'axes': [{'name': 'c', 'type': 'channel'}, {'name': 'z', 'type': 'space', 'unit': 'micrometer'}, {'name': 'y', 'type': 'space'}, {'name': 'x', 'type': 'space'}], 'datasets': [{'path': '0'}, {'path': '1'}, {'path': '2'}, {'path': '3'}], 'version': '0.3'}]
11:20:51 INFO root_attr: omero
11:20:51 DEBUG {'channels': [{'coefficient': 1, 'color': '00FFFF', 'family': 'linear', 'label': 'DAPI', 'window': {'end': 600, 'max': 65535, 'min': 0, 'start': 0}}, {'coefficient': 1, 'color': 'FF00FF', 'family': 'linear', 'label': 'nanog', 'window': {'end': 180, 'max': 65535, 'min': 0, 'start': 0}}, {'coefficient': 1, 'color': 'FFFF00', 'family': 'linear', 'label': 'Lamin B1', 'window': {'end': 1000, 'max': 65535, 'min': 0, 'start': 0}}], 'id': 1, 'name': 'TBD', 'version': '0.4'}
11:20:51 INFO datasets [{'path': '0'}, {'path': '1'}, {'path': '2'}, {'path': '3'}]
11:20:51 INFO resolution: 0
11:20:51 INFO  - shape ('c', 'z', 'y', 'x') = (3, 1, 19440, 20480)
11:20:51 INFO  - chunks =  ['1', '1', '2160', '2560']
11:20:51 INFO  - dtype = uint16
11:20:51 INFO resolution: 1
11:20:51 INFO  - shape ('c', 'z', 'y', 'x') = (3, 1, 9720, 10240)
11:20:51 INFO  - chunks =  ['1', '1', '2430', '2560']
11:20:51 INFO  - dtype = uint16
11:20:51 INFO resolution: 2
11:20:51 INFO  - shape ('c', 'z', 'y', 'x') = (3, 1, 4860, 5120)
11:20:51 INFO  - chunks =  ['1', '1', '2430', '2560']
11:20:51 INFO  - dtype = uint16
11:20:51 INFO resolution: 3
11:20:51 INFO  - shape ('c', 'z', 'y', 'x') = (3, 1, 2430, 2560)
11:20:51 INFO  - chunks =  ['1', '1', '2160 (+ 270)', '2560']
11:20:51 INFO  - dtype = uint16
11:20:51 DEBUG open(ZarrLocation(/tmp/xxx.zarr/B/03/0/labels))
11:20:51 DEBUG ZarrLocation.__init__ path:/tmp/xxx.zarr/B/03/0/labels, fmt:0.3
11:20:51 DEBUG Created nested FSStore(/tmp/xxx.zarr/B/03/0/labels, r, {'dimension_separator': '/', 'normalize_keys': False})
11:20:51 DEBUG ZarrLocation.__init__ /tmp/xxx.zarr/B/03/0/labels detected:FormatV03
11:20:51 DEBUG treating /tmp/xxx.zarr/B/03/0 [zgroup] as OMERO
11:20:51 INFO root_attr: multiscales
11:20:51 DEBUG [{'axes': [{'name': 'c', 'type': 'channel'}, {'name': 'z', 'type': 'space', 'unit': 'micrometer'}, {'name': 'y', 'type': 'space'}, {'name': 'x', 'type': 'space'}], 'datasets': [{'path': '0'}, {'path': '1'}, {'path': '2'}, {'path': '3'}], 'version': '0.3'}]
11:20:51 INFO root_attr: omero
11:20:51 DEBUG {'channels': [{'coefficient': 1, 'color': '00FFFF', 'family': 'linear', 'label': 'DAPI', 'window': {'end': 600, 'max': 65535, 'min': 0, 'start': 0}}, {'coefficient': 1, 'color': 'FF00FF', 'family': 'linear', 'label': 'nanog', 'window': {'end': 180, 'max': 65535, 'min': 0, 'start': 0}}, {'coefficient': 1, 'color': 'FFFF00', 'family': 'linear', 'label': 'Lamin B1', 'window': {'end': 1000, 'max': 65535, 'min': 0, 'start': 0}}], 'id': 1, 'name': 'TBD', 'version': '0.4'}
11:20:51 DEBUG creating lazy_reader. row:0 col:0 level:0
11:20:51 DEBUG creating lazy_reader. row:0 col:0 level:1
11:20:51 DEBUG creating lazy_reader. row:0 col:0 level:2
11:20:51 DEBUG creating lazy_reader. row:0 col:0 level:3
11:20:51 DEBUG treating /tmp/xxx.zarr/B/03 [zgroup] as ome-zarr
11:20:51 DEBUG returning /tmp/xxx.zarr/B/03 [zgroup]
11:20:51 DEBUG transforming /tmp/xxx.zarr/B/03 [zgroup]
11:20:51 DEBUG node.metadata: {'axes': [{'name': 'c', 'type': 'channel'}, {'name': 'z', 'type': 'space', 'unit': 'micrometer'}, {'name': 'y', 'type': 'space'}, {'name': 'x', 'type': 'space'}], 'name': ['DAPI', 'nanog', 'Lamin B1'], 'visible': [True, True, True], 'contrast_limits': [[0, 600], [0, 180], [0, 1000]], 'colormap': [[[0, 0, 0], [0.0, 1.0, 1.0]], [[0, 0, 0], [1.0, 0.0, 1.0]], [[0, 0, 0], [1.0, 1.0, 0.0]]]}
11:20:51 DEBUG Transformed: ([dask.array<from-value, shape=(3, 1, 19440, 20480), dtype=uint16, chunksize=(3, 1, 19440, 20480), chunktype=numpy.ndarray>, dask.array<from-value, shape=(3, 1, 9720, 10240), dtype=uint16, chunksize=(3, 1, 9720, 10240), chunktype=numpy.ndarray>, dask.array<from-value, shape=(3, 1, 4860, 5120), dtype=uint16, chunksize=(3, 1, 4860, 5120), chunktype=numpy.ndarray>, dask.array<from-value, shape=(3, 1, 2430, 2560), dtype=uint16, chunksize=(3, 1, 2430, 2560), chunktype=numpy.ndarray>], {'channel_axis': 0, 'name': ['DAPI', 'nanog', 'Lamin B1'], 'visible': [True, True, True], 'contrast_limits': [[0, 600], [0, 180], [0, 1000]], 'colormap': [<vispy.color.colormap.Colormap object at 0x7f1a61b62a60>, <vispy.color.colormap.Colormap object at 0x7f1a61b62af0>, <vispy.color.colormap.Colormap object at 0x7f1a61b62b80>]}, 'image')
11:20:51 DEBUG ImageSlice.__init__
11:20:51 DEBUG ImageSlice.__init__
11:20:51 DEBUG LOADING tile... 0/3
11:20:51 DEBUG ImageSlice.__init__
11:20:51 DEBUG LOADING tile... 0/0
11:20:51 DEBUG ImageSlice.__init__
11:20:51 DEBUG LOADING tile... 0/0
11:20:51 DEBUG ImageSlice.__init__
11:20:51 DEBUG ImageSlice.__init__
11:20:51 DEBUG ImageSlice.__init__
11:20:51 DEBUG ImageSlice.__init__
11:20:51 DEBUG ImageSlice.__init__
11:20:51 DEBUG ImageSlice.__init__
11:20:51 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:52 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG LOADING tile... 0/2
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG LOADING tile... 0/1
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG LOADING tile... 0/1
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG LOADING tile... 0/0
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG LOADING tile... 0/0
11:20:53 DEBUG ImageSlice.__init__
11:20:53 DEBUG LOADING tile... 0/0
11:20:54 DEBUG ImageSlice.__init__
11:20:54 DEBUG LOADING tile... 0/0
11:20:54 DEBUG ImageSlice.__init__
11:20:54 DEBUG LOADING tile... 0/0
11:20:54 DEBUG ImageSlice.__init__
11:20:54 DEBUG LOADING tile... 0/0
11:20:54 DEBUG ImageSlice.__init__
11:20:54 DEBUG LOADING tile... 0/0
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG LOADING tile... 0/0
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG LOADING tile... 0/0
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG LOADING tile... 0/1
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG LOADING tile... 0/2
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG ImageSlice.__init__
11:20:55 DEBUG ImageSlice.__init__
11:20:56 DEBUG ImageSlice.__init__
11:20:56 DEBUG ImageSlice.__init__
11:20:56 DEBUG ImageSlice.__init__
11:20:56 DEBUG ImageSlice.__init__
11:20:56 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__
11:20:57 DEBUG ImageSlice.__init__

[this example is for an image where the highest-resolution level has Channel-ZYX shape (3, 1, 19440, 20480)]

codecov · 2022-06-27T09:32:56Z

Codecov Report

Merging #209 (35cdce0) into master (313e126) will increase coverage by 0.03%.
The diff coverage is 90.90%.

@@            Coverage Diff             @@
##           master     #209      +/-   ##
==========================================
+ Coverage   83.86%   83.90%   +0.03%     
==========================================
  Files          12       12              
  Lines        1376     1379       +3     
==========================================
+ Hits         1154     1157       +3     
  Misses        222      222

Impacted Files	Coverage Δ
ome_zarr/reader.py	`85.00% <90.90%> (+0.11%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 313e126...35cdce0. Read the comment docs.

will-moore · 2022-06-30T16:49:19Z

This is working fine, (once I'd worked out how to allow Well viewing in napari ome/napari-ome-zarr#58 - apologies for the delay).

Opening a Well e.g.

$ napari --plugin napari-ome-zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/ -vvv

I initially see lower resolution tiles being loaded..

...
17:44:04 DEBUG LOADING tile... 2/1
17:44:04 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/2/1/.zarray
17:44:04 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/5/1/0.0.0.0.0
17:44:04 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/6/1/0.0.0.0.0
17:44:04 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/4/1/0.0.0.0.0
17:44:04 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/2/1/0.0.0.0.0
...

On zooming in, I see full resolution tiles being loaded..

17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/4/0/.zarray
17:45:21 DEBUG LOADING tile... 8/0
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/3/0/.zarray
17:45:21 DEBUG LOADING tile... 7/0
17:45:21 DEBUG LOADING tile... 6/0
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/2/0/.zarray
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/1/0/.zarray
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/5/0/.zarray
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/8/0/.zarray
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/7/0/.zarray
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/6/0/.zarray
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/6/0/0.0.0.0.0
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/4/0/0.0.0.0.0
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/7/0/0.0.0.0.0
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/8/0/0.0.0.0.0
17:45:21 DEBUG https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr/A/1/2/0/0.0.0.0.0

will-moore

Code looks good too. 👍
cc @sbesson

will-moore · 2022-06-30T16:56:41Z

ome_zarr/reader.py

-        column_count = math.ceil(math.sqrt(field_count))
-        row_count = math.ceil(field_count / column_count)
+        self.row_count = math.ceil(math.sqrt(field_count))
+        self.column_count = math.ceil(field_count / self.row_count)


Was there a reason to switch the order of column and row calculation here? Does it make any difference?

This was not supposed to slip into the PR, indeed.
I just took it out in c965afe.

FYI, The reason was that for a non-square well I was getting an 8x9 shape instead of the 9x8 shape I needed. But I'm not expecting the Well class to support non-square geometry in a reliable way, in its current implementation. If needed, I would rather make the required shape more explicit (e.g. by passing an optional grid_shape argument to Well).

sbesson · 2022-06-30T20:20:29Z

Would you suggest we release this as 0.6.0 as per the API additions?

will-moore

Great, thanks. LGTM

sbesson · 2022-07-01T08:56:50Z

Re-reading the details of the proposed changes, I convinced myself that the Well.get_field() and Well.get_lazy_well() methods are implementation details rather than public APIs e.g. as defined in the base Spec class.
Proposing to get this released later today with a patch version increment 0.5.1

…s (ref ome#209)

tcompa · 2022-07-01T09:03:35Z

Re-reading the details of the proposed changes, I convinced myself that the Well.get_field() and Well.get_lazy_well() methods are implementation details rather than public APIs e.g. as defined in the base Spec class.

Does 35cdce0 solve this issue?

EDIT: I only got it now, the point was about 0.5.1 vs 0.6.0, my bad. The question remains: would you rather have those two functions as Well methods or as part of its __init__?

will-moore · 2022-07-01T11:40:00Z

I don't have a strong opinion on the last commit, but slight preference for reducing the diff for this PR, so 👍 - and I re-tested the change and everything is still working.

sbesson · 2022-07-01T13:05:48Z

Thanks all this is now released as https://pypi.org/project/ome-zarr/0.5.1/

Read multi-resolution pyramids in Well class (closes ome#208)

63e3a92

tcompa mentioned this pull request Jun 27, 2022

Lazy loading of wells and multi-site plates fractal-analytics-platform/fractal-client#66

Closed

jluethi mentioned this pull request Jun 27, 2022

Read multiple fields of view on a grid, for HCS dataset #200

Open

will-moore mentioned this pull request Jun 28, 2022

Add minimal napari requirement for the plugin to be discovered ome/napari-ome-zarr#55

Merged

will-moore approved these changes Jun 30, 2022

View reviewed changes

Revert row_count<->column_count, in reader.py (ref ome#209)

c965afe

sbesson requested a review from will-moore July 1, 2022 08:10

will-moore approved these changes Jul 1, 2022

View reviewed changes

Move get_field and get_lazy_well from Well methods to simple function…

35cdce0

…s (ref ome#209)

sbesson merged commit fe811a6 into ome:master Jul 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Read multi-resolution pyramids in Well class (closes #208) #209

Read multi-resolution pyramids in Well class (closes #208) #209

tcompa commented Jun 27, 2022

codecov bot commented Jun 27, 2022 •

edited

Loading

will-moore commented Jun 30, 2022

will-moore left a comment

will-moore Jun 30, 2022

tcompa Jul 1, 2022

sbesson commented Jun 30, 2022

will-moore left a comment

sbesson commented Jul 1, 2022

tcompa commented Jul 1, 2022 •

edited

Loading

will-moore commented Jul 1, 2022

sbesson commented Jul 1, 2022

Read multi-resolution pyramids in Well class (closes #208) #209

Read multi-resolution pyramids in Well class (closes #208) #209

Conversation

tcompa commented Jun 27, 2022

codecov bot commented Jun 27, 2022 • edited Loading

Codecov Report

will-moore commented Jun 30, 2022

will-moore left a comment

Choose a reason for hiding this comment

will-moore Jun 30, 2022

Choose a reason for hiding this comment

tcompa Jul 1, 2022

Choose a reason for hiding this comment

sbesson commented Jun 30, 2022

will-moore left a comment

Choose a reason for hiding this comment

sbesson commented Jul 1, 2022

tcompa commented Jul 1, 2022 • edited Loading

will-moore commented Jul 1, 2022

sbesson commented Jul 1, 2022

codecov bot commented Jun 27, 2022 •

edited

Loading

tcompa commented Jul 1, 2022 •

edited

Loading