Write scale less than 5D #114

will-moore · 2021-09-21T09:29:03Z

Prompted by request for scripts to write masks data for napari: https://forum.image.sc/t/prepping-and-including-roi-masks/48750/17?u=will-moore

Since v0.3 now supports data with less than 5D, this should also be supported by the writing and scaling functionality.

This adds support for scaler.nearest(data, x, y) where data is 2D - 5D.
Same for write_image(data, scaler).

Breaking change
write_image() and write_multiscale() MUST be provided with axes argument (except for v0.1 or v0.2) since this is required by the v0.3 spec.

Also add 2D and 3D shapes to test_scaler and test_writer.

To test:

$ ome_zarr create coins
$ ome_zarr create --method astronaut astro

Then:

$ napari coins
$ napari astro

NB: this needs ome/napari-ome-zarr#15 for napari.

codecov · 2021-09-21T09:34:53Z

Codecov Report

Merging #114 (d65f5b3) into master (e68fc27) will decrease coverage by 0.28%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master     #114      +/-   ##
==========================================
- Coverage   70.39%   70.10%   -0.29%     
==========================================
  Files          11       11              
  Lines        1054     1104      +50     
==========================================
+ Hits          742      774      +32     
- Misses        312      330      +18

Impacted Files	Coverage Δ
ome_zarr/writer.py	`82.69% <70.83%> (-11.25%)`	⬇️
ome_zarr/data.py	`86.86% <87.87%> (-11.84%)`	⬇️
ome_zarr/scale.py	`63.71% <100.00%> (+3.13%)`	⬆️
ome_zarr/reader.py	`58.91% <0.00%> (+0.24%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e68fc27...d65f5b3. Read the comment docs.

will-moore · 2021-09-22T09:00:44Z

In response to the image.sc request above, I have tested this script locally:

# write_ome_zarr.py 

import numpy as np
import zarr
import os

from skimage.data import binary_blobs

from ome_zarr.io import parse_url
from ome_zarr.writer import write_image, write_multiscale


path = "test_ngff_image"
os.mkdir(path)

def create_data(shape, dtype, mean_val=10):
    rng = np.random.default_rng(0)
    return rng.poisson(mean_val, size=shape).astype(dtype)

size_z = 100
shape = (size_z, 256, 256)
data = create_data(shape, np.uint8)

blobs = binary_blobs(length=256, volume_fraction=0.1, n_dim=3)
blobs = blobs[:size_z, :, :]
print('blobs', blobs.shape)

# write the image data
store = parse_url(path, mode="w").store
root = zarr.group(store=store)
write_image(image=data, group=root, chunks=(1, 128, 128), axes="zyx")

# write the labels to /labels
labels_grp = root.create_group("labels")
# the 'labels' .zattrs lists the named labels data
label_name = "blobs"
labels_grp.attrs["labels"] = [label_name]
label_grp = labels_grp.create_group(label_name)
write_multiscale([blobs], label_grp, axes="zyx")

After running that script, I can view in napari:

$ napari test_ngff_image

After playing with rendering settings:

NB: also works in 3D view

joshmoore · 2021-09-22T15:03:35Z

Made it through a first reading and some minor local testing. (Looks like I need to update my local installation. I'm getting black planes in napari!)

One quick thought: would it make sense to add your blob() code to data.py as another example (perhaps the default)?

will-moore · 2021-09-22T15:52:31Z

Hmmm - I hadn't actually tested all the examples in data.py.

>>> from ome_zarr.data import create_zarr
>>> create_zarr("coins")

$ napari coins
Traceback (most recent call last):
  File "/opt/anaconda3/envs/napari/lib/python3.9/site-packages/napari/layers/utils/stack_utils.py", line 134, in split_channels
    i_kwargs[key] = next(val)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/anaconda3/envs/napari/bin/napari", line 8, in <module>
    sys.exit(main())
  File "/opt/anaconda3/envs/napari/lib/python3.9/site-packages/napari/__main__.py", line 424, in main
    _run()
  File "/opt/anaconda3/envs/napari/lib/python3.9/site-packages/napari/__main__.py", line 308, in _run
    viewer = view_path(  # noqa: F841
  File "/opt/anaconda3/envs/napari/lib/python3.9/site-packages/napari/view_layers.py", line 169, in view_path
    return _make_viewer_then('open', args, kwargs)
  File "/opt/anaconda3/envs/napari/lib/python3.9/site-packages/napari/view_layers.py", line 119, in _make_viewer_then
    method(*args, **kwargs)
  File "/opt/anaconda3/envs/napari/lib/python3.9/site-packages/napari/components/viewer_model.py", line 914, in open
    self._add_layers_with_plugins(
  File "/opt/anaconda3/envs/napari/lib/python3.9/site-packages/napari/components/viewer_model.py", line 987, in _add_layers_with_plugins
    added.extend(self._add_layer_from_data(*_data))
  File "/opt/anaconda3/envs/napari/lib/python3.9/site-packages/napari/components/viewer_model.py", line 1061, in _add_layer_from_data
    layer = add_method(data, **(meta or {}))
  File "/opt/anaconda3/envs/napari/lib/python3.9/site-packages/napari/components/viewer_model.py", line 757, in add_image
    layerdata_list = split_channels(data, channel_axis, **kwargs)
  File "/opt/anaconda3/envs/napari/lib/python3.9/site-packages/napari/layers/utils/stack_utils.py", line 140, in split_channels
    data_shape=data.shape,
AttributeError: 'list' object has no attribute 'shape'

will-moore · 2021-09-22T16:59:29Z

When I go into 3D view mode, the image and the labels align together, then when I return to 2D mode I see:

Can't see what's wrong here. Any ideas @joshmoore ?
Shapes of all the layers look good, as do all the .zarray blocks etc.

…arr()

will-moore · 2021-09-24T16:39:49Z

To test:

>>> from ome_zarr.data import create_zarr, astronaut
>>> create_zarr("coins")
>>> create_zarr("astro", astronaut)

Then:

$ napari coins
$ napari astro

NB: this needs ome/napari-ome-zarr#15 for napari.

imagesc-bot · 2021-09-25T21:44:46Z

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/prepping-and-including-roi-masks/48750/20

will-moore · 2021-10-05T10:58:31Z

That last commit should fix the issue that Khaled saw with the coins image initially rendered white in napari.
Settings for the coins image are now 0-255 instead of 0-1.

sbesson

A few inline comments, probably outside the scope of this PR. Overall the proposed changes look reasonable to me and in agreement with the 0.3 specification. Once this has been reviewed across the various use cases (synthetic data, OMERO Zarr export, napari), I would propose to get this out as 0.2.0.

sbesson · 2021-10-06T10:20:24Z

ome_zarr/writer.py

@@ -19,22 +19,57 @@ def write_multiscale(
    group: zarr.Group,
    chunks: Union[Tuple[Any, ...], int] = None,
    fmt: Format = CurrentFormat(),
+    axes: Union[str, List[str]] = None,


Possibly not an action for this PR but a general thought: as the specification gets refined and new concepts gets introduced (thinking concretely of the ongoing transformation proposal), there might be a trade-off between adding every new key as an extra parameter vs e.g. passing some form of dictionary of extra metadata which will be validated depending on the specification.

sbesson · 2021-10-06T10:21:19Z

ome_zarr/writer.py

    """

+    dims = len(pyramid[0].shape)
+    if fmt.version not in ("0.1", "0.2"):


While the axes guessing below definitely applies to 0.3, if the assumptions gets further relax, there will be an outstanding TODO of restricting this to the relevant versions of the specification.

will-moore · 2021-10-11T09:00:08Z

The down-sampling of a 3D array in the form of 3D labels was tested by @pwalczysko at ome/omero-cli-zarr#82 (comment)

will-moore · 2021-10-11T13:26:10Z

@khaledk2 Yes, that is expected. Zarr won't overwrite the existing data that you created with the first export.
You can either delete before repeating, or output to a different directory - see https://github.com/ome/omero-cli-zarr#usage

khaledk2 · 2021-10-11T14:33:49Z

I have tried
ome_zarr create coins
and then
napari coins
But it displayed a blank whight screen, although, it displayed what it should display when using the
ome_zarr create --method astronaut astro
and then
napari astro
I am using these two prs
#114
ome/napari-ome-zarr#15
I am not sure if I have missed something?

will-moore · 2021-10-11T14:59:40Z

@khaledk2 Can you try with the latest commit of this branch d65f5b3 which should fix the 'white' rendering setting.
If you installed with pip install -e . from you local ome-zarr-py repo, then you only need to fetch and update to that commit for your python env to use the latest version.

khaledk2 · 2021-10-12T08:56:51Z

@will-moore I have tested and it is working fine.

sbesson · 2021-10-12T09:42:06Z

Thanks all. Merging and going through the process of releasing ome-zarr 0.2.0

will-moore added 3 commits September 21, 2021 10:14

writer and scaler support less than 5D data

be4c85f

Tidy and fix scaling

6a3736f

Fix scaler.local_mean() for 2D-4D. Add 3D shape to tests

82aac73

will-moore added 2 commits September 21, 2021 12:05

write_image() MUST provide axes for v0.3

5485fef

Add axes in failing tests

982e360

data.py includes color for single-channel images

045a14e

will-moore added 3 commits September 23, 2021 16:28

Don't need axes for 5D data

438b0ab

data.create_zarr() doesn't enforce 5D data for v0.3

a509da9

Update test_ome_zarr.py to expect different shapes from data.create_z…

b038431

…arr()

This was referenced Sep 28, 2021

Use nested chunk store for labels ome/omero-cli-zarr#82

Merged

Fix failure to create Colormap for single-channel data ome/napari-ome-zarr#15

Merged

sbesson requested review from pwalczysko and khaledk2 October 1, 2021 14:23

ome_zarr create coins uses window 0-255 rendering

d65f5b3

sbesson reviewed Oct 6, 2021

View reviewed changes

sbesson merged commit 8bc556c into ome:master Oct 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write scale less than 5D #114

Write scale less than 5D #114

will-moore commented Sep 21, 2021 •

edited

Loading

codecov bot commented Sep 21, 2021 •

edited

Loading

will-moore commented Sep 22, 2021

joshmoore commented Sep 22, 2021

will-moore commented Sep 22, 2021

will-moore commented Sep 22, 2021

will-moore commented Sep 24, 2021 •

edited

Loading

imagesc-bot commented Sep 25, 2021

will-moore commented Oct 5, 2021

sbesson left a comment

sbesson Oct 6, 2021

sbesson Oct 6, 2021

will-moore commented Oct 11, 2021

will-moore commented Oct 11, 2021

khaledk2 commented Oct 11, 2021

will-moore commented Oct 11, 2021

khaledk2 commented Oct 12, 2021 •

edited

Loading

sbesson commented Oct 12, 2021 •

edited

Loading

Write scale less than 5D #114

Write scale less than 5D #114

Conversation

will-moore commented Sep 21, 2021 • edited Loading

codecov bot commented Sep 21, 2021 • edited Loading

Codecov Report

will-moore commented Sep 22, 2021

joshmoore commented Sep 22, 2021

will-moore commented Sep 22, 2021

will-moore commented Sep 22, 2021

will-moore commented Sep 24, 2021 • edited Loading

imagesc-bot commented Sep 25, 2021

will-moore commented Oct 5, 2021

sbesson left a comment

Choose a reason for hiding this comment

sbesson Oct 6, 2021

Choose a reason for hiding this comment

sbesson Oct 6, 2021

Choose a reason for hiding this comment

will-moore commented Oct 11, 2021

will-moore commented Oct 11, 2021

khaledk2 commented Oct 11, 2021

will-moore commented Oct 11, 2021

khaledk2 commented Oct 12, 2021 • edited Loading

sbesson commented Oct 12, 2021 • edited Loading

will-moore commented Sep 21, 2021 •

edited

Loading

codecov bot commented Sep 21, 2021 •

edited

Loading

will-moore commented Sep 24, 2021 •

edited

Loading

khaledk2 commented Oct 12, 2021 •

edited

Loading

sbesson commented Oct 12, 2021 •

edited

Loading