SLIC Layer: Superpixel

[old-school-kid]

Image Augment layer using SLIC
https://ieeexplore.ieee.org/document/6205760 cited by 7880
Implementation in skimage https://github.com/scikit-image/scikit-image/blob/v0.19.0/skimage/segmentation/slic_superpixels.py#L110-L385

Sharpen images by using a unsharp mask or something better I am unaware of.

[LukeWood]

Hey @old-school-kid any chance I could get a little more context as to how people tend to use this in the CV domain? Particularly w/ deep models.

Thanks

[old-school-kid]

Hi @LukeWood
SLIC isn't used in any CV model for image augmentation, afaik. But I have seen a lot of work using that as a pre-processing layer in my domain (material science) and even in medical imaging. In material science it helps in grain boundary detection and finding hairline fractures.
Do you want me to put links to some papers here using the preprocessing techniques? TIA.

[LukeWood]

That would actually be great. I'm really interested to see how it's used in both material science and medical imaging!

Thanks @old-school-kid

[LukeWood]

Also, would you be interested in contributing this @old-school-kid ?

[old-school-kid]

In Material Science

• Aluminum alloy microstructural segmentation Citation- 8
• Detection of microstructures in low carbon steel Citation- 8
• Classification of Bainitic Sub-Classes in Steel

In Medical Imaging

• A supervoxel‐based segmentation method for prostate MR images Citation- 36
• Detection and diagnosis of colitis on computed tomography using deep convolutional neural networks Citation- 40
• Segmentation and objective assessment of atrial scars Citation- 34

In Object Detection (Old methods, pre-2017)

• Salient object detection with high-level prior based on Bayesian fusion
• Salient object detection based on global multi-scale superpixel contrast
• Salient object detection method for breast tumor in ultrasound images based on absorbing Markov chain

[old-school-kid]

Also, would you be interested in contributing this @old-school-kid ?

Sure, would love to.

[bhack]

What is the final goal in your material science domain? Semantic segmentation? Or something different?

[bhack]

E.g. I am taking a look at https://github.com/Scientific-Computing-Lab-NRCN/MLography but I don't know if we still need some intermediate preprocessing like SLIC before the network learning stack.

[old-school-kid]

@bhack
Mostly Semantic segmentation, yes.

E.g. I am taking a look at https://github.com/Scientific-Computing-Lab-NRCN/MLography but I don't know if we still need some intermediate preprocessing like SLIC before the network learning stack.

I went through the repo. They have directly fed the image to an UNET which is fine, but generally we go for noise reduction and clustering (to separate matrix and grains) and this has proven to achieve better results.
This repo throws more light on this.

[bhack]

You repo is a little bit old and I am not an expert in the material science domain. But I remember that a few year ago, at ECCV 2018, there was a paper proposing a learning/differentiable approach ((currently 118 citations):

https://varunjampani.github.io/ssn/

But also if it was differentiable I don't know how much it is still popular today learning this intermediate representation.

[LukeWood]

@old-school-kid I'm fairly sure this is a good fit and is something that we'd be interested in hosting as long as the contribution is well written & maintainable.

So if you're interested please prepare a PR.

[old-school-kid]

@bhack

You repo is a little bit old and I am not an expert in the material science domain. But I remember that a few year ago, at ECCV 2018, there was a paper proposing a learning/differentiable approach ((currently 118 citations):

https://varunjampani.github.io/ssn/

That was a nice article. Thank you for sharing! While that can be used as a intermediate representation in end-to-end learning, SLIC is just used as a pre-processing layer and in no way it is fused with learning. But the article you shared looks promising too.

But also if it was differentiable I don't know how much it is still popular today learning this intermediate representation.

The papers I have shared above under Material Science are from 2019, 2020, 2021. Moreover apart from research, it is used in industry as it can easily segment matrix in microstructures.

[bhack]

Moreover apart from research, it is used in industry as it can easily segment matrix in microstructures

I think that this part could be a little bit out of the scope if not strictly functional to the learning step for this specific library.

But I still think that it is valid for a generic CV library (e.g. PIL, scikit image, Opencv, etc..).

[LukeWood]

This is in scope. The real question is whether or not the impact is high enough to justify maintaining the layer.

Medical image segmentation is an important domain with a lot of promise. SLIC is clearly showing promise there, and in other image segmentation areas.

This is especially in scope if the SLIC layer is differentiable. Then we can also use it as a preprocessing layer, or as it's used in https://varunjampani.github.io/ssn/.

The cost of maintaining a specialized layer like this is pretty low. So I am comfortable accepting it if @old-school-kid is willing to contribute a well refined implementation. If you do decide to work on it, feel free to add me as the reviewer on the PR 👍 .

[bhack]

In the last part of @old-school-kid comment It was quite clear that he mentioned that It Is useful also as is, without considering if It is useful or not in the learning pipeline.
For that specific part of the comment I think, but I could be wrong as I don't fully understand the Keras-cv policy, that we are not going to collect CV operations as is if they are not functional to the learning step or to populate metrics/visualizations.

Also more in general what I claimed Is that I think, just by a popularity/resources ratio point of view that this intermediate rappresentation, also when differentiable, It isn't not so popular anymore.

Instead if we are evaluating the non differentiable version I think that It could be useful to review this as a baseline in the context of modern proposed solutions for unsupervised/self-supervised/semi-supervised image segmentation papers (e.g. https://arxiv.org/abs/2007.09990 and other works).

[innat]

Implementation Ref.
https://albumentations.ai/docs/api_reference/augmentations/transforms/#albumentations.augmentations.transforms.Superpixels

[LukeWood]

Lets close this our as stale for now until theres a strong use case. If we end up trying to tackle a segmentation competition and can't compete without this, we should reprioritize.