introduce StructArrays

[johnnychen94]

We only have native support for an array of struct layout, the struct of array layout uses channelview/colorview but it losses the colorant information.

We might want to introduce StructArray to do some internal memory layout shifting without affecting the interface; it's still array of colorant but storing in a struct of array layout.

img = testimage("mandril")
out = StructArray(img)

out  == img # true

Of course, need some more sophiscate benchmark to decide this.

[kimikage]

One concern is that we do not recommend direct access to the fields of the AbstractRGB types.

[johnnychen94]

One concern is that we do not recommend direct access to the fields of the AbstractRGB types.

There are some justified use cases for this (e.g., filtering, per-channel algorithms), if StructArray simplifies the development without loss of performance then we should advertise it and perhaps providing native support inside JuliaImages.

I'm also considering hyperspectral images where pixel doesn't necessarily have traditionally colorant information. @maxfreu wondering if you have tried this package before?

[kimikage]

@Tokazama might also be interested in this...?
xref: JuliaImages/Images.jl#908

[kimikage]

WRT the interoperability between planar and packed (interleaved) formats, I was looking for a good way to handle chroma subsampling.

Chroma subsampling requires more sophisticated processing than StructArray, since it is not limited to a one-to-one correspondence between components and pixels. Chroma subsampling has many minor variations, so "I” have abandoned comprehensive support for it. However, it would be nice if it was supported (in a separate package from ImageCore.jl).

[Tokazama]

There are some really nice things about StructArray, but there are also some limitations that I don't see going away.
Perhaps it's enough to put a quick line in the docs on how to do this conversion by hand, because it should be a couple lines of code at the most.

[johnnychen94]

There are some really nice things about StructArray, but there are also some limitations that I don't see going away.

Would you mind explaining this a bit more? I'm not sure what specific limitations you're talking about. The unavoidable extra memory allocation, or something else?

[maxfreu]

@maxfreu wondering if you have tried this package before?

Yes, for rearranging storage layout of particles in 3D space in an MD sim. I can imagine how it could help e.g. with efficient transformations of keypoints in augmentor, but I do not yet see how it helps with multispectral images. But I'm clearly lacking experience here.

[timholy]

I'm also considering hyperspectral images where pixel doesn't necessarily have traditionally colorant information.

ColorTypes.jl was designed with this possibility in mind; in particular, the @makealpha macro was designed to allow you to add alpha-channel variants to custom types. For a particular camera you presumably know the characteristics of the hyperspectral output, and hence one could define

struct MyHSCamera <: Color{N0f16,8}
    channeldata::NTuple{8,N0f16}
end

and implement a few traits, and you'd theoretically be good to go.

[Tokazama]

I'm mostly referring to allocations, but it also depends on a lot of the Tables ecosystem. The only benefit this provides over MappedArrays is that it allows you to access each channel using getproperty syntax.

[timholy]

I've also noticed performance gaps between reinterpretc(RGB{T}, PermutedDimsArray(img, (3, 1, 2))) and a more direct implementation of HWC images. StructArrays might be able to help there, not sure.