Effects change pixels outside of selection

[novhack]

Pixelorama version:
Pixelorama master

OS/device including version:
Arch, AMD RX5700 XT - mesa 21.2.5

Issue description:
It seems that some Image shader effects like desaturation affect pixels with alpha<255 outside of selection even if "Only affect selection" is toggled.

I think that the problem is somewhere in ShaderImageEffect.gd because I encountered it also while trying to use it for bucket fill shader.

[dphaldes]

The issue here are lines 64-68 in the desaturate shader. It alters the output color's alpha based on the alpha values ignoring the selection texture.

[novhack]

This bug is the same in these effects:

• Invert Colors
• Destaturation
• Adjust Hue/Saturation/Value

[mrtripie]

If you do desaturation/HSV with 0 effect, and check what happened to the colors, you'll see that the semi-transparent colors got darker by exactly the amount of their alpha. So something to do with Premultiply Alpha is happening.

Adding this line to the shader fixes the effect once applied to the project, but has the opposite effect in the preview (where the semi-transparent pixels are now brighter):
render_mode blend_premul_alpha;

2021-12-19.13-27-04.mp4

According to this: godotengine/godot#17574 it seems to be due to how rendering images in viewports work.

I'm guessing what's happening is that the viewport gives us a texture that has its alpha premultiplied (making these pixels darker). The render_mode blend_premul_alpha on the shader is meant for displaying textures that have had their alpha premultiplied, and will make the semi-transparent pixels brighter. So what happens with this fix is that the semi-transparent pixels become brighter, then darker. (Which is the opposite of the usual way but I guess it works).

To fix the preview a couple things can be done:
1: Render the preview with ShaderImageEffect (which uses the viewport) to be consistent.
2: Put a "Multiply Alpha" shader on the preview TextureRect node to make the pixels darker again (There's already the shader there, can't work, unless there's a multiply alpha param/condition on the color adjustment shader)
3: Maybe have the previewing actually happen in the canvas using ShaderImageEffect (there could maybe be one of those before after sliders to compare)

---

There also seems to be another bug where some colors come out a little different, as shown in the blue (but only in the selection). I'm guessing this has to do with how the shader converts between RGB to HSV and back.