SSIM Alternative: DISTS

[torridgristle]

I'm trying DISTS as an alternative to SSIM and so far it's... It works to make the supplied image show up in the results. I don't know a whole lot about it though, to be frank it's just something different that works. I haven't decided if it's of any worth yet.

To get it running in Aphantasia I just needed to !pip install dists-pytorch and in clip_fft.py add from DISTS_pytorch import DISTS and replace ssim_loss = ssim.SSIM(window_size = 11) with ssim_loss = DISTS(require_grad=True, batch_average=True)

The two images to compare might need to be blurred before comparison to focus more on shapes.

[eps696]

thanks for info, let me know if you find it better than SSIM (i'm not planning to investigate such details soon).

good point about blurring as well; it should probably be done before applying that structure loss. i believe that CLIP comparison must be done with original image to keep all important features intact.

[torridgristle]

When I set it to recreate an image using just SSIM, no encoding whatsoever, it tends to look really smoothed out but the edges are there. However, when I tried it with DISTS it let all these neat shapes remain.

Also, it turns out earlier it wasn't even using clip_fft.py in the notebook, I had to put it in the Generate code cell and figure out why it kept complaining about all the tensors not being on the same device, only to realize that the loss apparently wasn't.

To actually get it working, had to add from DISTS_pytorch import DISTS and change ssim_loss = ssim.SSIM(window_size = 11) to ssim_loss = DISTS().cuda() and change ssim_loss(img_out, img_in) to ssim_loss(img_out, img_in, require_grad=True, batch_average=True), though it started off giving higher values than expected, like 0.55 or so, so I decided to square the output of it and also changed how it's used to calculate loss for testing, with

ssim_loss_output = ssim_loss(img_out, img_in, require_grad=True, batch_average=True)**2.0
loss = loss * (1.0 - sync) + ssim_loss_output * sync

which, at least for this, works well.

It doesn't overly match the image, but the overall general shapes are familiar.


with a sync of 0.5 that doesn't change as steps increase.

And I've been experimenting with a touch of weight decay in Adam in an attempt to break up big areas of nothing. Doesn't seem to help with that, doesn't seem to hurt, but it does make it wiggle and jiggle a bit in the final video which is really dreamlike, especially when the loss is turned up.

Aphantasia_DISTS_Test.zip
Here's my Colab notebook with it all set up. Had to manually !wget the weights for DISTS and move it, somehow the install didn't handle that on its own.

[eps696]

i tested DISTS and didn't find it convincing. besides quite fuzzy output, it tends to eat GPU RAM way more than SSIM.
what i used instead, was your idea about blurring: simple 8x downsampling before SSIM removed sharp edges and delivered exactly that level of details which i've been seeking (see below). so thanks for putting me up to this, and best of luck with further investigations.

text input = "winter", 200 steps, 240 samples, image input:

sync = 0.5

sync = 1.0

[russelldc]

@eps696 Not sure if you noticed, but this SSIM change (8x downsample) introduced a grid of tiny white or black pixels across the whole image. I was going to share my own example, but if you look closely at the 2 images you just posted, you should be able to see it.

[eps696]

sure noticed, they go with time (200-300 iters was enough for me)

[russelldc]

Odd, the grid doesn't seem to ever fade away in my experience. I've done several 1000+ iteration generations this afternoon, and it seems like it's a permanent effect.

[eps696]

well, few dots stay permanent indeed (like on the pics above), but their grid disappeared pretty quickly on all my tests. so i neglected this in favour of kinda proper reconstruction (any other trick didn't make it).
[i tried to fix it with additional blurring, but then details and edges of original image started appearing again, so i kept it like this]

[eps696]

just to fix the issue here: SSIM loss is now replaced by LPIPS, which allows way more delicate image integration, doesn't change colors (like SSIM), doesn't introduce ugly artefacts (like DISTS), and doesn't require downsampling (which introduced b/w dots). hopefully it will stay like that.
@torridgristle @russelldc