LCD Color Filter option is broken (crushed gradients and color shift)

[Azurfel]

Please fill out this form:

Expected behavior

The LCD Color Filter option should make the 240p Test Suite Grey ramp test pattern looks like this (screengrab of the standalone nightly 2.1.4-a42e166e from 2022-11-28 using the equivalent option "LCD Filter" taken using Win+PrntScn):

Actual behavior

On the VBA-M core (2.1.3-7e30b03), the LCD Color Filter option always looks like this:

The Screen capture menu option in the standalone with LCD Filter enabled client produces a more or less identically broken image:

Squall-Leonhart said that this is because "Captures go through a quantisation process for 256 color palettes, with the lcd filter enabled there is not enough bits left to cover all the grey shades, causing some to appear to be bluish green, others reddish."

No idea if that info is potentially useful here, or if the issue being so similar is pure coincidence.

Steps to reproduce the behavior

1. Generate a Grey ramp test pattern using the 240p test suite
2. Enable LCD Filter option

Crash bugs

N/A

ROM details

Effects all games.

Option details

Video driver (simple, opengl): effects all

Build details

Operating System (Windows, Mac, Linux (state distribution), etc.): Win10 21H1

Version of code (pre-built binary version, or commit ref, or just "master"): 2.1.3-7e30b03, along with a few other older versions.

Hardware details:

CPU: Intel 9700K
GPU: Nvidia RTX 2080
RAM: 32.0 GB

[negativeExponent]

libretro cores, by default uses 16bit colors for video framebuffer, which losses color information during post-processing (like how LCD filter, frame blending is done). here is a core compiled in 32bit colors with LCD filter enabled

not sure if its whats its suppose to look like. if its not, it might be better to post this on upstream since the filters used are the ones that upstream provided.

here is 32bit color without LCD filter

[Azurfel]

Your first picture with the LCD Color Filter on using the core compiled in 32bit colors looks perfect (it exactly matches the output of standalone nightly 2.1.4-a42e166e with the LCD Filter enabled, as well as the Gameboy Colors option from a rather old 2010 VisualBoyAdvance-M version i still have).