Transformation matrix with skimage.transform.warp

[fjorka]

Thank you for this great plugin!

I would like to reuse the saved transformation matrix from Affinder but after using it with skimage.transform.warp I get slightly different results. Could you help me figure out what I'm missing?

Napari 0.4.15
Affinder 0.2.2
skimage 0.19.2

[jni]

Hello @fjorka!!! Am I seeing you in a few weeks in SF???

So the issue here (I think) is that skimage.transform.warp, for historical reasons, uses x/y coordinate conventions, while napari uses NumPy/row-column coordinates. You can read more about NumPy coordinates here:

https://scikit-image.org/docs/dev/user_guide/numpy_images.html#coordinate-conventions

Yes, that's the scikit-image guide, but not all of scikit-image has been ported to use the new conventions, 😞 and warp still uses the old style. You can fix this two ways:

1. Use scipy.ndimage.affine_transform instead of skimage.transform.warp; or
2. Transpose the 0th and 1st row and column of the transformation matrix, then use it with warp:

def matrix_rc2xy(affine_matrix):
    swapped_cols = affine_matrix[:, [1, 0, 2]]
    swapped_rows = swapped_cols[[1, 0, 2], :]
    return swapped_rows

I think that should be enough to get the matrix to work with scikit-image...

Big picture, we intend to change this in skimage2, so that it will be compatible with NumPy coordinates. But that's like 1y from now so maybe try one of the two workarounds above in the meantime! 😅

[fjorka]

Hi! I'm definitely going to be in SF - can't wait to finally meet everyone in person!

Thanks for looking into it! The problem is actually more subtle. The Affinder transformation works with scikit-image and needs to be flipped for scipy - no problem with that. The problem is that aligned image as displayed in napari/affinder is slightly different from what I get from both skimage and scipy (and these are identical). This is a problem because my users will click points till they get something perfect as displayed by Affinder but then it will always be worse when used outside.

I tried to organize it in an example: https://github.com/fjorka/alignment_test/blob/master/affinder_test.ipynb
There is a screenshot there from what I see in napari:
magenta - Affinder correction
green - skimage
red - scipy
And you can see that green==red!=magenta. In this particular example it looks like scaling but it varies with the transformation.

I can give you an example reproducible without clicking if you help me with this issue from https://github.com/jni/affinder/blob/main/examples/basic-example.py

KeyError: "Plugin 'affinder' does not provide a widget named 'Start affinder'"

[jni]

Hey @fjorka,

I can give you an example reproducible without clicking if you help me with this issue

Yeah so that is because the example has changed since we moved to npe2. You can install the development version by cloning this repo, or you can just pip install -U affinder because I just pushed up a new release so that the example works with the released version.

I tried to organize it in an example

Thanks for this! It helped me to play with it seriously. The issue is that both ndimage.affine_transform and skimage.transform.warp expect the inverse transformation matrix, that is, the transformation from the reference image to the moving image. That's because when you want to create a new image, you need to find a value for every target pixel, so you want to go from every new pixel coordinate to the place it came from in the image you're transforming.

It just so happens that in this case, the inverse and the row/column transpose are close to each other. This might be generally true of affine transformation matrices but I'm not sure. So it looked like the values were close but actually it was a nonsense matrix. At any rate, here's code that shows it working:

from skimage import data, transform
from scipy import ndimage as ndi
import napari
import numpy as np

image0 = data.camera()
image1 = transform.rotate(image0[100:, 32:496], 60)

viewer = napari.Viewer()
l0 = viewer.add_image(image0, colormap='bop blue', blending='additive')
l1 = viewer.add_image(image1, colormap='bop purple', blending='additive')

qtwidget, widget = viewer.window.add_plugin_dock_widget(
        'affinder', 'Start affinder'
        )
widget.reference.bind(l0)
widget.moving.bind(l1)
widget()

viewer.layers['image0_pts'].data = np.array([[148.19396647, 234.87779732],
                                             [484.56804381, 240.55720892],
                                             [474.77521025, 385.88403205]])
viewer.layers['image1_pts'].data = np.array([[150.02534429, 80.65355322],
                                             [314.75696913, 375.13825634],
                                             [184.33085012, 439.81718637]])

def matrix_rc2xy(affine_matrix):
    swapped_cols = affine_matrix[:, [1, 0, 2]]
    swapped_rows = swapped_cols[[1, 0, 2], :]
    return swapped_rows

mat = np.asarray(l1.affine)
tfd_ndi = ndi.affine_transform(image1, np.linalg.inv(mat))
viewer.add_image(tfd_ndi, colormap='bop orange', blending='additive')
tfd_skim = transform.warp(image1, np.linalg.inv(matrix_rc2xy(mat)))
viewer.add_image(tfd_skim, colormap='bop orange', blending='additive', visible=False)

napari.run()

[jni]

I'm going to leave this open as a reminder to add this to the documentation.

[fjorka]

Hey @jni, and now it works beautifully! :) I was so confused by how close to the correct solution the swapped matrix was. I really appreciate your help - thank you!!!