Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend the axes fields in multiscales metadata #57

Merged
merged 18 commits into from
Jan 27, 2022
Merged

Extend the axes fields in multiscales metadata #57

Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4703d89
Start extending the axes related fields in multiscales
constantinpape Sep 7, 2021
e72ee48
Move axes to its own section
constantinpape Oct 1, 2021
212b2dd
Remove left over axes description from multiscales
constantinpape Oct 1, 2021
df835f1
Remove empty type; not necessary for SHOULD req
constantinpape Oct 1, 2021
b3523c9
Make number of dim requirement more explicit
constantinpape Oct 1, 2021
70ebede
Change axes to list of dicts
constantinpape Oct 18, 2021
a8597d7
Add transformation spec with only simple transformations from #63
constantinpape Oct 18, 2021
8ca7976
Clarify transformation order
constantinpape Oct 18, 2021
77790b3
Simplify transformations for multiscales
constantinpape Oct 27, 2021
755eaf1
Define list of transformations instead of a single transformation
constantinpape Oct 28, 2021
fa390f6
Add null as default value for axes:type
constantinpape Oct 28, 2021
47620ed
Add restrictions on number of axes and axis order to multiscale metad…
constantinpape Oct 28, 2021
ffcb2a8
Clarify ordering of spatial axes
constantinpape Nov 4, 2021
e27bdec
Try to fix rendering of table
constantinpape Nov 4, 2021
0661115
Add suggested units (copied from comment by @will-moore)
constantinpape Nov 25, 2021
23be956
Fix typo
constantinpape Nov 25, 2021
0419ce1
Add 'type' to the axes definition
constantinpape Jan 26, 2022
e94a0c1
Clarify the reference object for each new paragraph in the multiscale…
constantinpape Jan 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 82 additions & 24 deletions latest/index.bs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ Images {#image-layout}

The following layout describes the expected Zarr hierarchy for images with
multiple levels of resolutions and optionally associated labels.
Note that the number of dimensions is variable between 2 and 5 and that axis names are arbitrary, see [[#multiscale-md]] for details.
For this example we assume an image with 5 dimensions and axes called `t,c,z,y,x`.

```
. # Root folder, potentially in S3,
Expand All @@ -127,7 +129,7 @@ multiple levels of resolutions and optionally associated labels.
│ │ # by the "multiscales" metadata, but is often a sequence starting at 0.
│ │
│ ├── .zarray # All image arrays must be up to 5-dimensional
│ │ # with dimension order (t, c, z, y, x).
│ │ # with the axis of type time before type channel, before spatial axes.
│ │
│ └─ t # Chunks are stored with the nested directory layout.
│ └─ c # All but the last chunk element are stored as directories.
Expand Down Expand Up @@ -205,49 +207,105 @@ Metadata {#metadata}
The various `.zattrs` files throughout the above array hierarchy may contain metadata
keys as specified below for discovering certain types of data, especially images.

"axes" metadata {#axes-md}
--------------------------

"axes" describes the dimensions of a physical coordinate space. It is a list of dictionaries, where each dictionary describes a dimension (axis) and:
- MUST contain the field "name" that gives the name for this dimension. The values MUST be unique across all "name" fields.
constantinpape marked this conversation as resolved.
Show resolved Hide resolved
- SHOULD contain the field "type". It SHOULD be one of "space", "time" or "channel", but MAY take other values for custom axis types that are not part of this specification yet.
- SHOULD contain the field "unit" to specify the physical unit of this dimension. The value SHOULD be one of the following strings, which are valid units according to UDUNITS-2.
- Units for "space" axes: 'angstrom', 'attometer', 'centimeter', 'decimeter', 'exameter', 'femtometer', 'foot', 'gigameter', 'hectometer', 'inch', 'kilometer', 'megameter', 'meter', 'micrometer', 'mile', 'millimeter', 'nanometer', 'parsec', 'petameter', 'picometer', 'terameter', 'yard', 'yoctometer', 'yottameter', 'zeptometer', 'zettameter'
- Units for "time" axes: 'attosecond', 'centisecond', 'day', 'decisecond', 'exasecond', 'femtosecond', 'gigasecond', 'hectosecond', 'hour', 'kilosecond', 'megasecond', 'microsecond', 'millisecond', 'minute', 'nanosecond', 'petasecond', 'picosecond', 'second', 'terasecond', 'yoctosecond', 'yottasecond', 'zeptosecond', 'zettasecond'

If part of [[#multiscale-md]], the length of "axes" MUST be equal to the number of dimensions of the arrays that contain the image data.


"transformations" metadata {#trafo-md}
-------------------------------------

"transformations" describes a series of transformations, e.g. to map discrete data space of an array to the corresponding physical space.
It is a list of dictionaries. Each entry describes a single transformation and MUST contain the field "type".
The value of "type" MUST be one of the elements of the `type` column in the table below.
Additional fields for the entry depend on "type" and are defined by the column `fields`.

| type | fields | description |
| ------------- | ------ |------------ |
| `identity` | | identity transformation, is the default transformation and is typically not explicitly defined |
| `translation` | one of: `"translation":List[float]`, `"path":str` | translation vector, stored either as a list of floats (`"translation"`) or as binary data at a location in this container (`path`). The length of vector defines number of dimensions. |
| `scale` | one of: `"scale":List[float]`, `"path":str` | scale vector, stored either as a list of floats (`scale`) or as binary data at a location in this container (`path`). The length of vector defines number of dimensions. |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at how this table is rendered at http://api.csswg.org/bikeshed/?url=https://raw.githubusercontent.com/ome/ngff/47620eddae0d04a078505ae7189b7e0e94f5feed/latest/index.bs#trafo-md somethings not right

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just needs at least 3 --- below the header: https://www.markdownguide.org/extended-syntax/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking. I haven't looked at the rendering at all yet; will give it another look.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@will-moore I have tried to fix this. How do I find the link for the latest deployment to check?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The URL is in the README. Basically just replace the checksum e.g. http://api.csswg.org/bikeshed/?url=https://raw.githubusercontent.com/ome/ngff/e27bdecc06dedc71c6ec7e20a2bd7dd0755d43ea/latest/index.bs#trafo-md

I think I failed to get this to work which is by the version table at the bottom uses <table>.


In addition, the field "axisIndices" MAY be given to specify the subset of axes that the transformation is applied to, leaving other axes unchanged. If not given, the transformation is applied to all axes. The length of "axisIndices" MUST be equal to the dimensionality of the transformation. If "axisIndices" are not given, the dimensionality of the transformation MUST be equal to the number of dimensions of the space that the transformation is applied to.
If given, "axisIndices" MUST be given in increasing order. It uses zero-based indexing.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a lot of rules to handle the case where some axes don't need a scale.
I wonder if instead of
"scale": [0.5, 0.5, 0.5], "axisIndices": [2, 3, 4] we could do "scale": [null, null, 0.5, 0.5, 0.5]. Then we can simply say that the length of scale list must be equal to the number of dimensions. And we don't need to have all the axisIndices rules (and validation logic). It's also less code to find the scale for e.g. dimension 2.

scale_dim2 = transform["scale"][2]

compared with:

index = transform["axisIndices"].indexOf(2)
if index == -1:
  scale_dim2 = None
else:
  scale_dim2 = transform["scale"][index]

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@will-moore - how would you know that those are the spatial axes? while a certain ordering is recommended it is not mandatory under the spec? the spec currently decouples dimension from type of axes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree that the above case is shorter, but the code to generate a 3d -> 3d transform would be longer with the
[null, null, 0.5, 0.5, 0.5] case. I.e. the code would have to check which dimensions are spatial, then slice the scale array appropriately.

I don't have much of a preference between the two, honestly. But some things are easier if the transforms always consider dimensions of the same type.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"scale": [null, null, 0.5, 0.5, 0.5] is simply an easier way to store the info that is in "scale": [0.5, 0.5, 0.5], "axisIndices": [2, 3, 4]. In both cases the spatial axes are 2, 3 and 4 (because they're not null in [null, null, 0.5, 0.5, 0.5].
If you ONLY had spatial axes. e.g. zyx then it's simply "scale": [0.5, 0.5, 0.5].
If the dimension order is different. e.g. z, t, y, x then use "scale": [0.5, null, 0.5, 0.5] instead of "scale": [0.5, 0.5, 0.5], "axisIndices": [0, 2, 3]

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need null for scale --- you can just use 1. And I agree that axisIndices just adds unnecessary complexity.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bogovicj Can you explain a bit more why it is so important to be able to specify axisIndices?

Do you imagine frequently writing out this metadata fully manually?

With the current state of this proposal, it appears that affine transforms aren't even supported --- only translation and scale. So there it is just a matter of specifying 0 or 1 (for translation or scale, respectively) for any axes that are to be left alone --- it seems like axisIndices would really only provide a benefit when manually writing out the metadata for a full affine transform (not yet part of the proposal, but presumably planned to be added).

You mentioned the specific use case of translating x and y only. For the use case you have in mind, is the translation by an integer amount? If so, would that use case perhaps be better addressed by support for a non-zero origin in zarr?
zarr-developers/zarr-specs#122

For programmatic reading and writing of this metadata, a single affine transform would seem to be the most straightforward way to deal with these transformations. I suppose implementations that don't support anything but translation and scaling might instead normalize to a single translation vector and single scale vector. But they could just as well use an affine transform but return an error if the rotation matrix is not diagonal. It seems to me that composing of multiple transforms, and support for things like axisIndices, as specified in the current proposal, could instead by handled by library functions in the program used to generate the OME-zarr metadata.

Likewise, in a graphical user interface a single affine transform would also seem to be the most straightforward way to present the transformation, as otherwise there is a lot of added complexity in displaying possibly multiple different transforms of different types, and making it clear in what order they are applied. In Neuroglancer affine transforms are shown in their normal matrix form but with the rows and columns labeled to indicate to which source/target dimension they correspond and which column corresponds to translation. That makes it fairly easy to understand even if you have more than 3 dimensions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bogovicj Sorry, I didn't appreciate how this would work with affine transforms. I agree that your example with axisIndices looks simpler, although I don't actually understand either!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To quickly revive this conversation: as @bogovicj explains, we have introduced axisIndices to enable specifying transformations for a subset of the axes, e.g. for the use-case of affine transformations.
I think having the option of specifying subsets for transformations is crucial to have in the transformations, but I agree that axisIndices may not be the best solution. I see a few potential options:

  1. change to using named axis for all the transformation parameters, e.g. scale={"x": 2, "y": 2} (as discussed above). Names must be the same as given in the axes definition. In this case we don't need axisIndices
  2. just remove axisIndices from the current proposal and deal with how to address subset of axes later.
  3. keep everything as is and improve how axisIndices are documented

My personal preference would be option 1, because I think that this would be the most elegant solution.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the nice summary @constantinpape .

I like option (1) as well.

please don't make me create a 5D non-linear transform to move only the x-y axes around.

I need to clarify this, so let me describe a simpler example as a case study. There is a fairly common use case in which we need to apply a nonlinear 2D transform to every plane of a 3D volume (e.g., lens correction).

This is not covered an offset or an affine. We like the axisIndices (or similar) idea here because it lets us store the transform in a straightforward way: as a 2D displacement field (a 3D volume: [2,X,Y], where the 2 dimension is a vector describing the displacements at [:,x,y]).

Without axisIndices (or some way to communicate "this transform is applied over a subset of the coordinates), then what is the right way to store the transform / communicate this idea? I guess by making it a 3D displacement field instead? So a [3,X,Y,Z] array where the [2,:,:,:] is full of zeros, and [0:1,:,:,z] all contain identical data. That's wasteful.

I'm not sure how to deal with this use case without being explicit about this "subset idea".

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would absolutely agree that having to store extra zeros would be unacceptable, and agree that something like axisIndices is very useful in the flow field case.

You could accomplish the same thing, in a somewhat more general way, by allowing to specify an affine transform that is applied to every flow vector. However, I'm not sure if that extra generality would actually be helpful.


The transformations in the list are applied sequentally and in order.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sequentially

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to specify precisely the mathematical formula in terms of transformed space coordinates and original coordinates. Otherwise there is a possibility for confusion as to the precise interpretation of the scale and translation values.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comment @jbms. I think this is a good point.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just trying to implement the handling of scale and translation in napari-ome-zarr. Napari supports scale and translate metadata (see add_image). Napari doesn't specify an order, but it seems by testing that the translate values are in real-world values (applied after scaling).
Do we want to specify the same for translation in this NGFF spec? Or does this depend on the ordering of translation and scale? If the translation comes before scale then it's in pixels. If after then it's in real space?
Also, just a thought with naming. In napari it's scale and translate (both verbs) but we have scale and translation. Do we want to use translate instead?



"multiscales" metadata {#multiscale-md}
---------------------------------------

Metadata about the multiple resolution representations of the image can be
found under the "multiscales" key in the group-level metadata.
Metadata about an image can be found under the "multiscales" key in the group-level metadata. Here, image refers to 2 to 5 dimensional data representing image or volumetric data with optional time or channel axes. It is stored in a multiple resolution representation.

"multiscales" contains a list of dictionaries where each entry describes a multiscale image.

Each dictionary contained in the list MUST contain the field "datasets", which is a list of dictionaries describing
the arrays storing the individual resolution levels.
Each "multiscales" dictionary MUST contain the field "axes", see [[#axes-md]].
The length of "axes" must be between 2 and 5 and MUST be equal to the dimensionality of the zarr arrays storing the image data (see "datasets:path").
The "axes" MUST contain 2 or 3 entries of "type:space" and MAY contain one additional entry of "type:time" and MAY contain one additional entry of "type:channel" or a null / custom type.
The order of the entries MUST correspond to the order of dimensions of the zarr arrays. In addition, the entries MUST be ordered by "type" where the "time" axis must come first (if present), followed by the "channel" or custom axis (if present) and the axes of type "space".
If there are three spatial axes where two correspond to the image plane ("yx") and images are stacked along the other (anisotropic) axis ("z"), the spatial axes SHOULD be ordered as "zyx".
will-moore marked this conversation as resolved.
Show resolved Hide resolved
The values of the "name" fields must be given as a list in the field "_ARRAY_DIMENSIONS" in the attributes (.zattr) of the zarr arrays.
This ensures compatibility with the [xarray zarr encoding](http://xarray.pydata.org/en/stable/internals/zarr-encoding-spec.html#zarr-encoding).
E.g. for "axes: [{"name": "z"}, {"name": "y"}, {"name": x}]", the zarr arrays must contain "{"_ARRAY_DIMENSIONS": ["z", "y", "x"]}" in their attributes.

Each "multiscales" dictionary MUST contain the field "datasets", which is a list of dictionaries describing the arrays storing the individual resolution levels.
Each dictionary in "datasets" MUST contain the field "path", whose value contains the path to the array for this resolution relative
to the current zarr group. The "path"s MUST be ordered from largest (i.e. highest resolution) to smallest.

It MUST contain the field "axes", which is a list of dimension names of the axes.
The values MUST be unique and one of `{"t", "c", "z", "y", "x"}`.
The number of values MUST be the same as the number of dimensions of the arrays corresponding to this image.
In addition, the "axes" values MUST be repeated in the field "_ARRAY_DIMENSIONS" of all scale groups
(i.e. groups containing arrays with the multiscale data).
This ensures compatibility with the [xarray zarr encoding](http://xarray.pydata.org/en/stable/internals/zarr-encoding-spec.html#zarr-encoding).
Each "datasets" dictionary MUST have the same number of dimensions and MUST NOT have more than 5 dimensions. The number of dimensions and order MUST correspond to number and order of "axes".
Each dictionary MAY contain the field "transformations", which contains a list of transformations that map the data coordinates to the physical coordinates (as specified by "axes") for this resolution level.
constantinpape marked this conversation as resolved.
Show resolved Hide resolved
The transformations are defined according to [[#trafo-md]]. In addition, the transformation types MUST only be `identity`, `translation` or `scale`.
They MUST contain at most one `scale` transformation per axis that specifies the pixel size in physical units.
It also MUST contain at most one `translation` per axis that specifies the offset from the origin in physical units.
If both `scale` and `translation` are given `translation` must be listed after `scale` to ensure that it is given in physical coordinates. If "transformations" is not given, the identity transformation is assumed.
The requirements (only `scale` and `translation`, restrictions on order) are in place to provide a simple mapping from data coordinates to physical coordinates while
being compatible with the general transformation spec.

It SHOULD contain the field "name".
Each "multiscales" dictionary MAY contain the field "transformations", describing transformations that are applied to each resolution level.
The transformations MUST follow the same rules about allowed types, order, etc. as in "datasets:transformations".
These transformations are applied after the per resolution level transformations specified in "datasets". They can for example be used to specify the `scale` for a dimension that is the same for all resolutions.

It SHOULD contain the field "version", which indicates the version of the
multiscale metadata of this image (current version is 0.3).

It SHOULD contain the field "type", which gives the type of downscaling method used to generate the multiscale image pyramid.
Each "multiscales" dictionary SHOULD contain the field "name". It SHOULD contain the field "version", which indicates the version of the multiscale metadata of this image (current version is 0.4).

Each "multiscales" dictionary SHOULD contain the field "type", which gives the type of downscaling method used to generate the multiscale image pyramid.
It SHOULD contain the field "metadata", which contains a dictionary with additional information about the downscaling method.

```json
```
{
"multiscales": [
{
"version": "0.3",
"version": "0.4",
"name": "example",
"datasets": [
{"path": "0"},
{"path": "1"},
{"path": "2"}
],
"axes": [
"t", "c", "z", "y", "x"
{"name": "t", "type": "time", "unit": "millisecond"},
{"name": "c", "type": "channel"},
{"name": "z", "type": "space", "unit": "micrometer"},
{"name": "y", "type": "space", "unit": "micrometer"},
{"name": "x", "type": "space", "unit": "micrometer"}
],
"datasets": [
{
"path": "0",
"transformations": [{"type": "scale", "scale": [0.5, 0.5, 0.5], "axisIndices": [2, 3, 4]}] # the voxel size for the first scale level (0.5 micrometer)
}
{
"path": "1",
"transformations": [{"type": "scale", "scale": [1.0, 1.0, 1.0], "axisIndices": [2, 3, 4]}] # the voxel size for the second scale level (downscaled by a factor of 2 -> 1 micrometer)
},
{
"path": "2",
"transformations": [{"type": "scale", "scale": [2.0, 2.0, 2.0], "axisIndices": [2, 3, 4]}] # the voxel size for the second scale level (downscaled by a factor of 4 -> 2 micrometer)
}
],
"transformations": [{"type": "scale", "scale": [0.1], "axisIndices": [0]], # the time unit (0.1 milliseconds), which is the same for each scale level
"type": "gaussian",
"metadata": { # the fields in metadata depend on the downscaling implementation
"method": "skimage.transform.pyramid_gaussian", # here, the paramters passed to the skimage function are given
Expand Down