Transformation Specification

[constantinpape]

This is a feature request towards v0.2 of the multi-scale image spec:
Support (affine) transformations for images to apply these on the fly (e.g. to map images to a shared coordinate system).

So far, I have come around two different implementations that contain a proposal for this:

Open Organelle

{
"datasets": [
      {
        "path": "s0",
        "transform": {
          "axes": ["z", "y", "x"],
          "scale": [1, 1, 1],
          "translate": [0, 0, 0],
          "units": ["nm", "nm", "nm"]
         }
       }
   ]
} 

The transformation metadata is also duplicated in the individual datasets.

I2K Tutorial

Same as the open organelle proposal, but the transformation is defined for on the dataset level, not for each individual scale:

{
"datasets": [...],
"transform": {...}
}

The transformation metadata is not duplicated.

(I have left out the nested multiscale etc. in both cases to make this more readable.)

I would be happy to develop this into a PR to the spec, but it would be good to have some feedback first:

• Is there a competing approach we should take into account? Any general critique?
• Should the transformation be defined per dataset or per scale level (I think per scale level actually makes more sense).
• Do we add rotation and shear to transformation, so that a full affine trafo can be specified?

@joshmoore @frauzufall @kephale @axtimwalde @d-v-b @tischi @jni
(please tag anyone else who might be interested in this)

---

Original text from #12:

Originally discussed as part of the multiscales specification, scale metadata related to each volume in a multiscale image needs storing for the proper interpretation of the data.

see also ome/omero-iviewer#359 in that each level should be strictly smaller than the last.

[jni]

Note that there is a scale "discussion" ;) in #12.

Some considerations:

• "axes" appears to be the same as "dimension_order", or at least intimately related, as discussed during the meeting. This is probably not the right place for this information?
• In napari, we're now finding that many users want to specify scale/translate in one place, and an affine transformation in addition to this. I thought coming in to this issue that it was mostly about the latter bit...?
• From my perspective, and with utmost respect to affine transforms 😂, I think scale/translate/physical units is priority 1, and affine should come later. Having affine and scale/translate in the same discussion complicates it unnecessarily (do you represent affine as a matrix or as the decomposed values? How do you describe rotations in 3D? etc)
• With regards to axis order etc, I think (?) the original OME model uses things like PhysicalSizeX and PhysicalSizeY, which I hope we can avoid in ngff but I dunno, it sounds like compatibility to existing OME metadata is a concern @joshmoore?

[constantinpape]

Note that there is a scale "discussion" ;) in #12.

Ok, good point I did not see that one and it's indeed related.

* "axes" appears to be the same as "dimension_order", or at least intimately related, as discussed during the meeting. This is probably not the right place for this information?

Yes, axes is the dimension order as it refers to the values for this transformation. One could also store this at a higher level in the metadata and then assume that the transform values always refer to this axis order.

• In napari, we're now finding that many users want to specify scale/translate in one place, and an affine transformation in addition to this. I thought coming in to this issue that it was mostly about the latter bit...?

• From my perspective, and with utmost respect to affine transforms joy, I think scale/translate/physical units is priority 1, and affine should come later. Having affine and scale/translate in the same discussion complicates it unnecessarily (do you represent affine as a matrix or as the decomposed values? How do you describe rotations in 3D? etc)

So the idea here would be to describe one transformation per image. I think chaining transformations is currently out of scope.
Regarding affine vs. translation / scale: Sorry, I should have expanded this more, but I thought about the following design:
Transform has the following 4 fields:

"scale": [],
"translate": [],
"rotate": [],
"shear": []

They are all optional, where their absence indicates a scale of 1, no translation, no rotation etc.
This way the full affine transformation can be specified but one can also just specify the scale, translation etc.
Then indeed one would need to agree on how to describe the rotation and shear (euler angles for the rotation?).
With the MoBIE hat (and I think also BDV hat) on, it would be pretty important to be able to describe the complete affine right away.

[d-v-b]

"axes" appears to be the same as "dimension_order", or at least intimately related, as discussed during the meeting. This is probably not the right place for this information?

I decided to put "axes" in there because I work alongside people who use languages / libraries that use different array indexing schemes, and so we ran into lots of annoyances due to ambiguity about axis names (is the first axis z, or is it x?). axes also allows you to convert the transform object into a key-value pair with meaningful keys (in fact, I considered a key-value pair with semantic keys for the transform object.). And I think this information does belong with scale and translate, since it clarifies their meaning.

And I think every scale level should have the transform listed. The principle for that is "multiscale is just a collection of single-scale datasets => multiscale metadata is just a collection of single scale metadata".

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-feb-23/48386/9

[constantinpape]

And I think every scale level should have the transform listed. The principle for that is "multiscale is just a collection of single-scale datasets => multiscale metadata is just a collection of single scale metadata".

We modeled the metadata here after the open organelle layout, but at the time didn't see a need for a transformation for each scale level. But I have thought about this more since then, and I agree that it's better to have it for each scale level, which allows to display them separately.

[d-v-b]

I'm not 100% sure there is an absolute right answer here. One drawback of listing transforms in multiscales is that metadata gets duplicated -- the transform is specified once in dataset metadata, and again in multiscales, which is kind of ugly and could be a burden if transform metadata gets too big, or multiscales spatial metadata somehow differs from the dataset-level spatial metadata.

However, from an implementation standpoint, it's very convenient to access metadata from a single consolidated file (i.e., a single GET), rather than accessing it from multiple files (i.e., multiple GETs). This convenience was also a motivation for putting the the spatial transform stuff in multiscales, and the access pattern advantages outweighed worries about duplicating information. But maybe other people reach the opposite judgment.

[joshmoore]

Note that there is a scale "discussion" ;) in #12.

Ok, good point I did not see that one and it's indeed related.

And has basically nothing in it. More than happy to let @constantinpape take the lead. Closing #12.

[constantinpape]

One drawback of listing transforms in multiscales is that metadata gets duplicated -- the transform is specified once in dataset metadata

We didn't put it in the dataset metadata at all and only in multiscales. I think it's definitely benefitial to have it consolidated in multiscales to reduce the number of request and make it easier to access.

I don't feel strong about having it with the individual datasets or not; I think for what we design the viewer would always access the multiscales. This would be something for @joshmoore to weight in on how he feels about duplication of metadata in the spec.

[d-v-b]

I don't feel strong about having it with the individual datasets or not; I think for what we design the viewer would always access the multiscales. This would be something for @joshmoore to weight in how he feels about duplication of metadata in the spec.

I think it's critical that individual datasets always have spatial metadata. The multiscales metadata cannot replace the individual spatial metadata of a single dataset, because datasets should be portable -- I should be able to open a single dataset, outside of the context of a multiscale pyramid, and get valid spatial information.

[jni]

I should be able to open a single dataset, outside of the context of a multiscale pyramid, and get valid spatial information.

I agree very much with this design constraint. To me it certainly feels more important than the problem of metadata duplication, which can be managed.

[constantinpape]

Ok, then I think we have clear perference: transformation metadata must be specified in the individual datasets and in multiscales.
Now, we only need to define the actual trafo metadata :).

[axtimwalde]

What do you guys think of speccing out a flexible extensible trafo meta data format with clear exit points when something is not supported? Our very old approach to this is the TrakEM2/ Render format (http://www.ini.uzh.ch/~acardona/api/mpicbg/trakem2/transform/CoordinateTransform.html and https://github.com/saalfeldlab/render/blob/master/docs/src/site/markdown/data-model.md). The gist

transformation {type:String, data:String}

and

list{[transformation|list]}

The entire spec here is very Java centric (type is the legit class name), and data encode all sorts of stuff into a String which is limiting in how much you can map into it. Also, a reference to a file/ dataset (for deformation fields or coefficients) is not intended.
My heart would be a lot lighter if we could focus on generating such a spec including all transformations that we know of or have implementations of at this time instead of drooling over how to write parts of affine transformations.

[constantinpape]

@axtimwalde
If I understand your proposal correctly, some example transformations could look ike this:

{
  "transformation": {"type": "affineBdv", "data": "..."},  # affine transformation in bigdataviewer format
  "transformation": {"type": "eulerElastix", "data": "..."}  # euler transformation in elastix format
  ...  
}

and we would somewhere list the available types and the corresponding format for "data".

And list would be a list of these transformations that allows chaining them?

[axtimwalde]

Approximately, but the data : String approach is very limiting and it should therefore be flexible what to put into the parameters. Often, that would be something like

"transformation" : {"type" : "scale", "parameters" : [4, 4, 40]}

[constantinpape]

@axtimwalde I think this is a good solution, because it also makes it easier to extend the transformations later. (E.g. to support #30).
And it would allow to very simply specify scale and translation and concatenate other transformations, @jni.

I am also not so sure whether adding the axes and units fields (or similar) to the transformation is such a good idea. This metadata refers to the data and not specifically to the transformation. How would we deal with inconsistent values between the metadata for the transformation and the data?

Also, in the current form, the image spec always assumes 5D data. However, the transformations are only valid for the 2 or 3 spatial dimensions. This is something we need to keep in mind too, but I think we first need to discuss the 5D requirement more, see #35.

[d-v-b]

My heart would be a lot lighter if we could focus on generating such a spec including all transformations that we know of or have implementations of at this time instead of drooling over how to write parts of affine transformations.

I disagree :) I don't think trying to come up with a spec that covers all known / possible nonlinear transformations is feasible or a good use of time. In my view, as soon as you go down the road of applying a nonlinear coordinate transformation to your data, you are working with specialized tools and the developers of those tools should be allowed to use whatever metadata they need. This only requires that the ngff transformation spec be permissive.

On the other hand, I think it does make sense to be formal about linear coordinate transformations, because they are universal -- every image processing library understands affine transforms (and most visualization tools can display data on an affine-transformed grid, with varying levels of support for shear and rotation). Because linear transforms are universal, I think it does make sense to come up with (or copy) a readable, compact description of linear transforms. This could either be a list of named transformations, like scale and translate, or a full matrix. We just need to ensure that we include information that gives meaning to the rows and columns of this matrix.

I am also not so sure whether adding the axes and units fields (or similar) to the transformation is such a good idea. This metadata refers to the data and not specifically to the transformation. How would we deal with inconsistent values between the metadata for the transformation and the data?

I don't think axes and units describe the data; instead, axes and units describe the grid on which the data is sampled. The spatial transform under discussion also describes this grid. In particular for shears and rotations it's very important to explicitly name the dimensions of the input space and the dimensions of the output space -- in medical applications, images might be acquired in machine coordinates and transformed into anatomical coordinates.

[axtimwalde]

Not every image/ array processing library supports affine transformations, translations or rigid transformations have fewer parameters than affine transformations which justifies treating them differently than affine transformations, but default decomposing affine transformations makes little sense in practical applications that use affine transformations and is incredibly verbose. Axes in data space are usually a mixture of several physical dimensions, i.e. a scanner scans a spatial dimension while spending time, so the 'correct' axes label would be "x + t" with a unit of "0.4um + 0.0000s" or something. It is quickly clear that particularly the qualifiers in the unit field are clearly parts of a transformation. I therefore strongly believe that named axes with references to physical spaces and units should be used for the target space of a transformation, not the data/ source space. The data has axes in null or data space, it's default unit is "px". A transformation converts them into physical space with named axes and physical dimensions (if you want). A transformation can transform between all sorts of spaces, not only physical and data. I feel bad about muddling a "default" axes permutation (another subset of affine transformations) into the spec by naming the axes of the transformation, so they can be used for data with axes that have the same name. That is a very complicated way of specifying the axes permutation into a 'default' data space in which some other transformation can be applied. I think it would be great for transformations to have names, e.g. "xycz2xyzc" for one such axes permutation. A 'standard library' of transformations may only contain affine transformations and subsets thereof at first and may be extended by other classes of transformations and reference implementations as needed.

[constantinpape]

Thanks for the detailed comments @d-v-b @axtimwalde. I will try to go through it point by point:

Transformation Specification:

I don't think trying to come up with a spec that covers all known / possible nonlinear transformations is feasible or a good use of time. [...] developers of those tools should be allowed to use whatever metadata they need.

Not every image/ array processing library supports affine transformations, translations or rigid transformations have fewer parameters than affine transformations which justifies treating them differently than affine transformations, but default decomposing affine transformations makes little sense in practical applications that use affine transformations and is incredibly verbose.

I think we are on a very similar page here. The transformation spec

{"type": "some_transformation_type", "parameters": {...}}

allows to specify custom transformations. But to support interoperability, we would provide a list of "standard" transformation types (e.g. translation, scale, rigid, affine) with a strict definition. Developers can still use their custom transformations, but then there is no guarantee other tools can parse it. (At some point the list of standard types could be extended with common custom transformations, but let's not get ahead of ourselves here).

Axes and units:

I therefore strongly believe that named axes with references to physical spaces and units should be used for the target space of a transformation, not the data/ source space. The data has axes in null or data space, it's default unit is "px". A transformation converts them into physical space with named axes and physical dimensions

This makes a lot of sense. The only open question for me is what to do about image datasets that don't specify a transformation, but for which one still needs to specify the axes.
Potential solutions:

1. All image datasets must specify a transformation
2. Add additional metadata to specify the physical dimensions in case no transformation is given

I would prefer 1, but this would be a breaking change with the current spec.

Axes permutations:

I feel bad about muddling a "default" axes permutation (another subset of affine transformations) into the spec by naming the axes of the transformation, so they can be used for data with axes that have the same name. That is a very complicated way of specifying the axes permutation into a 'default' data space in which some other transformation can be applied.

I agree with this.

My point here was that it's unclear how to indicate which axes of the input data the transformation is applied to.
Note that in the current spec data is always 5 dimensional, which makes this necessary. If we drop the 5D requirement from the spec, see also #35, we could demand that transformation and data have the same number of dimensions and avoid this issue.

[d-v-b]

I therefore strongly believe that named axes with references to physical spaces and units should be used for the target space of a transformation, not the data/ source space. The data has axes in null or data space, it's default unit is "px". A transformation converts them into physical space with named axes and physical dimensions

If the axes of the source space are not named, then information is needlessly lost. Imagine point-scanning an image of a region in space, then scanning the same region after rotating the sample 90 degrees (i.e., transposing the scan axes). Without semantics for the source axes (e.g., names like scan_y, scan_x), it is cumbersome to distinguish these two images. But I agree that a default unit of px makes sense for the source space.

Potential solutions:

1. All image datasets must specify a transformation
2. Add additional metadata to specify the physical dimensions in case no transformation is given

I would prefer 1, but this would be a breaking change with the current spec.

I also prefer 1, and I note that option 2 is just shoehorning in a specification of a transform ("physical dimensions" is just a scale transform), so these two options aren't really different.

[tischi]

+1 for "all data sets must specify a transformation" (aka a mapping from data space to physical space)

As @constantinpape pointed out, to practically proceed further I guess the question of the dimensionality of the data space indeed seems of quite high priority.

[axtimwalde]

Thanks for picking this up @d-v-b and @constantinpape.

If the axes of the source space are not named, then information is needlessly lost. Imagine point-scanning an image of a region in space, then scanning the same region after rotating the sample 90 degrees (i.e., transposing the scan axes). Without semantics for the source axes (e.g., names like scan_y, scan_x), it is cumbersome to distinguish these two images.

This is a transformation. Rotating the sample will most likely not be a perfect 90deg rotation either, so being able to do this with a flexible transformation spec instead of renaming axes would be much desired.

1. All image datasets must specify a transformation

This is compatible with specifying no transformation: No transformation means identity transformation. Named axes and units are specified for the target space of the transformation. No named axes means data space with unit "px". This is compatible with named axes and units for images because that means that the target space is achieved by applying the identity transformation after which "px" are "cm". The current standard case of scaled named axes to express the resolution is also compatible with this, the scaling as the transformation, the axes names and units are the target space.

[tischi]

I am wondering how a transformation from, e.g., a 3D data space that should be mapped into physical space with the dimensions XYC would look like? Did you consider this or are we currently assuming that all dimensions of the transformation are always spatial? I think an argument for supporting a data space XYC was that some people have data with lots of channels and would like to configure the chunking such that all channels for one location XY are loaded as one chunk.

[tischi]

Let's say I want 5 micrometer voxel spacing in the physical space and translate 10 micrometer and have XYC data space. Would something like this be the idea?

affine = 
{ 5, 0, 0, 10,
  0, 5, 0, 10,
  0, 0, 1,  0 }
axisLabels: {"x", "y", "c"}
axisUnits: {"micrometer", "micrometer", "channel"}

[thewtex]

In napari, we're now finding that many users want to specify scale/translate in one place, and an affine transformation in addition to this. I thought coming in to this issue that it was mostly about the latter bit...?

👍 Since anisotropic pixel spacing and an offset is common with microscopy and other types of scientific images, supporting spacing and offset as standard spatial metadata associated with the image makes sense.

These two are built into an xarray.Dataset representation as discussed in #48.

A few other items for consideration:

Parameter precision turns out to be very important in practice with spatial transformations (lessons learned over the years in itk). Serialization and deserialization in decimal ascii in general will result in a loss of precision due to the differences in binary / decimal encoding and how IEEE float's are represented. This is most significant for transformations that involve rotations. In general, special effort has to be taken to avoid loss of precision, e.g. google double-conversion. So, it is best to store the transformation parameters in float64 arrays.

It works well to store a spatial transformation, identified by the transformation type and its parameters, along with an identifier of the coordinate system it transforms from and the coordinate system it transforms to. Then, the image holds in its metadata the identifier for its coordinate system. A few examples of this include DICOM spatial transformations 1 2 and WIP in NeurodataWithoutBorders (NWB).

Even for "simple" rigid transformations, there are a dozen ways that they can be interpreted unless all the information is there to interpret them. As @axtimwalde and I learned when examining the bioimage electron microscopy community prior to standardization, a dozen researchers will somehow manage to all pick the different ways 🥚 🍳 🍞 🥖 🥪

In terms of possible transformations, ITK has defined a nice phylogeny (attached).
SpatialTransformations.pdf

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/fiji-hackathon-2021-big-data-days-a/53926/7

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-early-september-2021/55333/14

[bogovicj]

Here is an early but concrete proposal @axtimwalde and I wrote based on the discussion above that reflects our preferences (of course).

https://github.com/saalfeldlab/n5-ij/wiki/Transformation-spec-proposal-for-NGFF

[constantinpape]

Here is an early but concrete proposal @axtimwalde and I wrote based on the discussion above that reflects our preferences (of course).

https://github.com/saalfeldlab/n5-ij/wiki/Transformation-spec-proposal-for-NGFF

Thanks, John! Overall I really like this proposal and its extensible and already covers some of the non-linear use-cases of #30.

A couple of comments:

• The Axis types section has diverged a bit from the current solution in v0.3, see Data dimensionality and axes metadata #35 (comment). But I think it's feasible to combine the current version and the proposal.
• How would you specify which axes the transformations act on, e.g. if we have axes tczyx and an affine? How would the software now that its applied to zyx? Related to this, it might not be a good idea to hard-code xyz axis order. Maybe adding a field that specifies which axes the trafo acts upon would solve this?
• For d_field and p_field we could also allow (or demand) storing the parameters as a zarr array in the same container, which would help in keeping all relevant data together.

In addition, following the discussions in #53, there seems to be strong support to specify a transformation per scale dataset. I think the easiest way to do this would be

"datasets": [
  {"path": "s0", "transform": {"type": "affine", "affine": [...]}},
  {"path": "s1", "transform": {"type": "affine", "affine": [...]}},
  ...
]

[tischi]

@bogovicj @axtimwalde

Thanks a lot for fleshing this out! ❤️

I also have a couple of questions:

Transformations are specified as forward transformations, i.e. a transformation transforms a vector from source into target space

What's the motivation to specify in this direction? For example I think in the elastix registration software they specify from target to source. I am not saying that I would prefer to specify it in the other direction, I am just curious!

Axes types

The axes right now have a field called label. In the case of a spatial axis, you gave an example where you gave the axis the label y. I wonder, is this (a) just some text, such that we also could have called it foo, or (b) the label contains information about how this axis should be "arranged" in the right-handed spatial coordinate system? Personally, I'd be OK with assigning the "labels" x,y,z, and t specific meanings, because I would say this is a very well established convention in many physics text-books and we live in a physical world. But I think, e.g., @jni had concerns about this?

[bogovicj]

Thanks for the feedback Constantin! @constantinpape

On axes:

I think it's feasible to combine the current version and the proposal.

I agree. The only "big" difference that I see to what you describe in #35 is the "type" (spatial, time, etc.). We might also see the nrrd spec. and as you say there:

it would be easy to have an equivalent solution using 3 lists, e.g. axes, axes_label and unit.

which I would be very happy with.

[pd]-fields

For d_field and p_field we could also allow (or demand) storing the parameters as a zarr array in the same container, which would help in keeping all relevant data together.

I think this is a great idea.

"which axis does my transform operate on"

How would you specify which axes the transformations act on

You've pinpointed a hard thing. I'm not sure, and am open to ideas.
Here's a wild collection of ideas
ranging from

"shove all the transformations together"

{
      "transform": {
        "type": "affine",
        "affine": [
            2.0, 0.0, 0.0, 10.0,
            0.0, 0.5, 0.0, -5.0,
            0.0, 0.0, 1.0, 0.0
        ]
      },
      "axisTypes": ["spatial","spatial","channel"],
      "axisLabels": ["X","Y","C"]
  }

to "split by type"

  "axes": [
      {
          "type":"spatial",
          "labels":["X","Y"],
          "indicies":[0,1],
          "transform": {
              "type":"affine",
              "affine":[2.0, 0.0, 10.0,
                        0.0, 0.5, -5.0 ]
          },
          "unit":"nm"
      },
      {
          "type":"channel",
          "labels":["C"],
          "indicies":[2],
          "transform": {
              "type":"identity"
          }
      }
  ]

the former is simpler, the latter "clearer" somehow and does not let one "mix" different types of axes. I'm even tempted to allow a couple of options ranging from simple to general.

[bogovicj]

Thanks @tischi !

transform inverses

What's the motivation to specify in this direction? For example I think in the elastix registration software they specify from target to source.

Exactly! Our thinking was that it fits most people's (? certainly my) mental model better. In the case of elastix output, we have in mind to do something like:

{
   "type" : "inverse_of",
   "transformation" : {
      "type" : "bspline_dfield",
      "data" : "the parameters..."
   }
}

axes

(a) just some text, such that we also could have called it foo, or (b) the label contains information about how this axis should be "arranged" in the right-handed spatial coordinate system?

Good point. I was thinking (a), kind of. More we mean it to be a way for people / software that generates data to encode something about it. I'm thinking about what you say here.
Sometimes the first index of the volume increasing is arbitrary (call it x or foo). Sometimes is not, call it anterior-posterior, A-P, or left-right / LR or whatever.

So, I wasn't thinking about (b), but maybe we should.

I'd be OK with assigning the "labels" x,y,z, and t specific meanings

me too.

[tischi]

So, I wasn't thinking about (b), but maybe we should.

Yes, I guess we need something to tell how the spatial axes are mapped onto the right-handedness, isn't it?

[tischi]

...or is that maybe implicit in the transformation, i.e. the right-handedness is a feature of sample space and not voxel space. Thus, maybe just the order of the spatial axes after the transformation defines the axes order and thereby the handedness (to this end, within the transformation one could "switch" axes if needed).

[d-v-b]

I'd be OK with assigning the "labels" x,y,z, and t specific meanings

What does this mean exactly? I think wherever possible the semantics of axis names should not play any role in this spec. The spec should only "know" that axes have string names. I think mapping string names on to real life stuff is the job of tools that consume data, not the storage layer.

[bogovicj]

I'd be OK with assigning the "labels" x,y,z, and t specific meanings

I understand this to mean: x,y,z are spatial dimensions and x cross y = z
t is a time dimension

I think mapping string names on to real life stuff is the job of tools that consume data, not the storage layer.

How can the consuming software know that some spatial dimension is left-right, vs right-left, vs inferior-superior if the storage layer doesn't tell it?

[d-v-b]

I understand this to mean: x,y,z are spatial dimensions and x cross y = z
t is a time dimension

If this kind of information is part of the spec, it creates interaction between axis names and axis types -- would it be a violation of the spec to have an axis called z that has type time? I think this kind of validation should be done by consuming software, not the storage layer.

How can the consuming software know that some spatial dimension is left-right, vs right-left, vs inferior-superior if the storage layer doesn't tell it?

If I understand it correctly, this kind of information pertains to the relationship between axes, not the mapping from axis names onto physical stuff, and so this seems in-scope for the spec.

[constantinpape]

I just had time to quickly read all the recent comments, but I think there is some need for clarification of the current (v0.3) axes field and its relation to the proposal:

• current: we have a single list axes, which must only contain the values tczyx, which all have a clear semantic meaning (t time, c channel, rest spatial). I.e. currently axis label and semantic meaning are tightly linked.
• the proposal contains 3 fields per axis: label (= free text), type (= semantic meaning of the axis, i.e. space, time or channel), unit (e.g. seconds, meters, etc.)

I think there are some good arguments to switch to something closer to the proposal and decouple label and semantic meaning. Ideally we would do this in a way that is somewhat compatible with v0.3 (i.e. the field axes with values in tczyx should still be valid).

I will read the rest of the comments during/after the meeting.

[constantinpape]

Follow up from last week's meeting:
I will work on a PR for transformations, based on the proposal by @bogovicj and @axtimwalde. This will likely be included in v0.4 of the spec. There will be a PR to extend the axes definition beforehand, see #35 (comment).

[thewtex]

For d_field and p_field we could also allow (or demand) storing the parameters as a zarr array in the same container, which would help in keeping all relevant data together.

I think this is a great idea.

Could we put all transform parameters in binary floating point .zarray's? This will help preserve precision and with compression.

[bogovicj]

Agreed @thewtex .

Added path fields to the spec that describe where in this container a transformation's parameters are stored as binary data.

[tischi]

@constantinpape looking at the current proposal from @bogovicj it is not clear to me whether it would be possible to specify different transformations for different channels.
Use case: data acquired with a microscope where two channels are acquired with separate slight misaligned cameras, leading to a channel shift, which could be accomodated by means of specifying a translation of one of the channels.

[bogovicj]

@tischi, I went crazy, let me know what you think.

You're right, the spec does not allow multiple transforms per dataset. I would personally split channels into different datasets if I was going to process / transform them differently.

However, what you want is possible by using (abusing?) a displacement field (and doesn't even cost lots of extra storage), but it either requires some assumptions or modifications. Maybe it's a little unclear / overkill though. I'm open to adding something to cover that case more clearly.

I assume N channels everywhere, here are some options:

One transform per dataset

The stuff below gets messy. We might declare v0.X handles exactly one transformation per dataset. I'm okay with this option.

Use a displacement field

We could do the above by using N displacement fields (stored in one dataset), but it's not a "field" so much because there's only one displacement per channel (assuming 3 channels). I'd argue "boundary" extension is the most sensible for displacement fields, so can be assumed. But we could be explicit about it either by spec-ing the extension

"shape" : [ 1, 1, N, 3 ],
"axes" : {
   "labels": ["x","y","c","v"],
   "type": ["space","space","c","vector"],
   "unit": ["ly","ly","","um"]
},
"extend" : [ "boundary", "boundary", "0", "0" ]

or abusing the transform (pixel width there is 1 billion tera-meters :p). I actually hate this.

"shape" : [ 1, 1, N, 3 ],
"axes" : {
   "labels": ["x","y","c","v"],
   "type": ["space","space","c","vector"],
   "unit": ["Tm","Tm","","um"]
},
"transform" : {
   "type" : "scale",
   "scale" : [1e9, 1e9, 1, 1]
{

The downside is that the displacement vectors are 3D to be explicit about which vector component applies to which dimension. We could go to shape [1,1,N,2], "displacements" : ["x","y"]

Coordinate-wise

Another way modify the spec to make this possible is to add a new "coordinate-wise" transformation that might look like this:

{
   "type" : "coordinate-wise",
   "axis" : "c",
   "indexes" : [ 2, 5 ],
   "transformations" : [
   	{ "type" : "translation",
   	  "translation" : [-0.5, 0.0]
   	},
   	{ "type" : "translation",
   	  "translation" : [0.0, 0.9]
   	}
   ]
}

which says "translate the channel at index 2 by -0.5 units, and translate the channel at index 5 by 0.9 units".
"indexes" will be optional with default [0,1,...N], where N is the length of the transformations list.

This would stink if there were lots and lots of channels, so let's make that less painful.

Transformation_list

A "transformation_list" stores many transformations of a single type, in a dataset.

{
  "type" : "coordinate-wise",
  "axis" : "c",
  "transformations" : {
  	"type": "transformation_list",
  	"member_type": "translation",
  	"path": "/path/to/channel_translations"
  }
}

the above requires a dataset in this container at "/path/to/channel_translations" of shape Nx2 (or 2xN?)
and defines N translations. But that's looking quite a lot like a displacement field, but without being explicit about which dimensions the translation applies to, but maybe not so bad.

Even more general?

If we allow even more general axis types, we go as far as describing a piece-wise affine with say:

"axes" : {
   "labels": ["x","y","c","matrix"],
   "type": ["space","space","c","affine_parameters"],
   "unit": ["um","um","c","um"]
}

but I think it's not worth going the far right now.

[tischi]

@bogovicj

Thanks a lot! I also had an idea (not sure if this is covered by your proposals, it feels a bit similar to your coordinate-wise proposal).

• We allow for several transformations.
• For each transformation one needs to specify on which axes it operates on.
• For all the axes that the transformation does not operate on one can specify a range of indices to which the transformation should be applied. The default is all indices.

What do you think?

By the way, another use case would be drift correction where one may want to specify different translations per timepoint.

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/ome-zarr-hackathon-january-2022/60771/1

[constantinpape]

Closing in favor of #94