-
-
Notifications
You must be signed in to change notification settings - Fork 35.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WebGLRenderer: Add support for AgX Tone Mapping #27366
Conversation
📦 Bundle sizeFull ESM build, minified and gzipped.
🌳 Bundle size after tree-shakingMinimal build including a renderer, camera, empty scene, and dependencies.
|
src/renderers/shaders/ShaderChunk/tonemapping_pars_fragment.glsl.js
Outdated
Show resolved
Hide resolved
src/renderers/shaders/ShaderChunk/tonemapping_pars_fragment.glsl.js
Outdated
Show resolved
Hide resolved
We can look at Filament as well, see google/filament#7236. It does reference the same article, perhaps they are effectively the same. Do we want to support the two looks, "Punchy" and "Golden", as well? These are represented in Blender by selecting the AgX view transform and then one of the looks. But if we want this in three.js, we could represent them simply as three enums. Perhaps:
I don't feel strongly whether we need the looks in this PR. But it might be nice to name the default something we'll still be happy with if we add them later.
In the past I've compared tone mapping implementations by creating a Linear-sRGB OpenEXR 'source' image, then running it through Blender's tone mappers using the OCIO CLI, and tone mapping it in three.js using EXRLoader and MeshBasicMaterial. The results should match exactly, other than very slight differences in our approximation and the LUTs. I'll share a sample 'source' image and reference renders from OCIO, a bit later. EDIT: There are these common reference images too: https://projects.blender.org/blender/blender/pulls/106355 ... I've seen the source images somewhere, but will have to dig around for those. |
…sl.js Co-authored-by: Don McCurdy <[email protected]>
…sl.js Co-authored-by: Don McCurdy <[email protected]>
On second thought, let's not add looks in this PR — Blender 4.0 stable does not represent them as it did in the alpha/beta releases, Golden is removed, and we may need to look closer at what has changed. |
Thanks! I think I'd prefer to avoid digging through their rendition of it especially If they're referencing the same implementation. But I'd definitely like to get some test images to make sure it matches the other implementations.
I'll skip adding them in this PR but even if we do long term I'm okay with
I saw at least one of these images in filament's PR, as well. If these are canonical test images it would be nice to know where they're from. Unfortunately I'm not all that fluent in Blender so making my own versions through Blender may be difficult. |
Here's the source, by the author of AgX: https://github.com/sobotka/Testing_Imagery Those are source files, mostly in Linear-sRGB. I'd stay out of this repository's subfolders. The images are not tone-mapped, but loading them with EXRLoader and setting |
Can you please explain why you increased the env map intensity? (I assume you felt it was necessary.) Tone mapping will not "fix" an over-exposed scene. |
I think it's useful here for illustration purposes — without tone mapping we see the "Notorious Six" begin to appear. Whether the image is over-exposed is a more subjective judgment, and I don't feel strongly without more context than an isolated model. If we limit our examples to only use exposure ranges that look nice without tone mapping, that would be a mistake. |
I did not mean to suggest we do either of those. Tone mapping requires the scene to be (at least reasonably) pre-exposed. It is not useful to compare tone mapping algorithms on a test case that is over-exposed. |
Agreed! But, I don't consider But I assume you are maybe trying to get at questions about the relative brightness of images produced with AgX and ACES Filmic, and whether we should make adjustments there as we did with ACES Filmic for the typical display environments of three.js users? I agree if so, and I think the OpenEXR test imagery will be helpful in that discussion. |
@donmccurdy I think we are concerned about the same things. :-) But I was particularly interested if @gkjohnson had a reason for setting the env intensity so high. Did he compensate for that by reducing exposure in his test case? |
It's as Don suggested - I felt it was more interesting to show the tone mapping performed with brighter values. No tonemapping exposure adjustment was done and it was increased by using envMapIntensity.
I never suggested this was the case. I'm not sure what you're meaning by this.
Thanks! I'll take a look at these later and use the Blender versions to compare. |
I reached out to Troy about AgX's intended viewing conditions. There's more to the thread that is useful, but a short summary of the immediately-relevant parts might be:
https://fosstodon.org/@donmccurdy/111570779516336527 In most AgX implementations, "looks" are exposed as convenient ways to customize contrast and other characteristics of the tone mapper. |
I'm seeing some differences between the comparison images here. I'll look through the code to make sure I didn't flub any of the port and other settings are flipped somewhere. Also @donmccurdy - that folder only seemed to contain one of the images from the Blender PR. Do you know where the others came from? It might also be helpful to have a few sample images from blender that we generate so we know all the settings used. Would it be possible to generate a few samples from the EXR images using Blender for this?
|
I think I've seen two of these (the bar scene and the underwater scene) before in AgX tests. Perhaps the rest are Blender's own tests, I'm not sure. In any case, we can run Blender's own color management transforms on the files from this repository. Steps on macOS: outdated shell commands, see later posts# Install OpenColorIO (OCIO) and OpenImageIO (OIIO).
brew install opencolorio openimageio
# Save path to Blender's OCIO config to an alias.
OCIO=/Applications/Blender.app/Contents/Resources/4.0/datafiles/colormanagement/config.ocio
# Standard (no tone mapping).
oiiotool --colorconfig "$OCIO" -i "blue_bar_709.exr" --colorconvert "Linear Rec.709" "sRGB" -o ./out/blue_bar_standard.png
# AgX, Medium Contrast.
oiiotool --colorconfig "$OCIO" -i "blue_bar_709.exr" --colorconvert "Linear Rec.709" "AgX Base sRGB" --ociolook "Medium Contrast" -o ./out/blue_bar_agx.png I don't know the color spaces of these images unless identified by the filename, so I'm limiting my tests to those with "Rec 709" or "BT 709" in the name. These are Linear-sRGB, for our purposes. [ invalid screenshots removed, see updated versions below ] |
The EXRs were attached to this post: https://projects.blender.org/attachments/8ab6aa53-418a-4d72-89be-cce7e36d3dd0 Also if you are looking at the EXRs in Blender, do it in the compositor, don't do it in EEVEE with emission plane. EEVEE clips the negatives. |
I've ported Filaments implementation in this PR, now, which converts our rec 709 Linear sRGB input colors to rec2020 Linear colors using the transform matrix from this paper, performs the AgX conversion, then converts back to rec 709. Generally I think the image looks better - as in the relative colors more closely reflect Blenders image and the color transitions on the cheek are smoother. The color is still a bit more washed out, though, which makes me think the Blender "guard rails" and luminance compensation are doing a good amount of work. @EaryChow Would that explain the differences in the Blender and Filament port images?
Is there any write up or documentation what the "guard rail" and luminance compensation are doing and why? From looking around at this post a bit I'm assuming no, but maybe that's changed. Unfortunately I'm not so familiar with python and associated libraries so a more abstract write up would be easier understand first if it's available. That said, though, it doesn't look like Filament implements the guard rail approach at all. |
Also, regarding the failed tests - does CI run on WebGL 1? The tests that are failing should work in WebGL 2. It says that a |
The PR in its current state indeed breaks any tone mapping usage in WebGL 1. I've tested that with At least up to Maybe you can use something like the following to support GLSL < 3.0.
|
@gkjohnson I think we've diverged on the references somehow. Your Blender reference image for "red_xmas_rec709" above and in #27366 (comment) doesn't match my own image from #27366 (comment) (see script at end of that post), perhaps others as well. In case my previous shell scripts — accidentally using Looks from older Blender Filmic — were leading things astray, I've pushed my script to a gist I can update more easily: https://gist.github.com/donmccurdy/ca1d7cbb107435b40fa510f783f91ee8 |
Thanks for the catch. I've been copy and pasting the image from your comment which seems to be changing the resulting color of the image for some reason - at least the reds are becoming brighter. I've updated both the previous comments to use the direct URLs from the previous comments to make sure the images are the same, now. The colors are still a bit washed out compared to the latest commit but less so. And an example of the color changing:
I can fix the PR when the results are good enough - I was just surprised to see that the CI was using WebGL 1. It seems like it could cause problems when WebGL1 support is dropped or cause rendering discrepancies. |
I have already posted the sorce code of Blender implementation:
I believe you haven't implemented any rotation of primaries as part of the inset yet, judging from your previous blue bar example. The result is that, previously with Linear Rec.709 working primaries, the Rec.709 R, G, B, three strips from the sweep will be attenuated along chromaticity-linear paths, resulting in things like Abney Effect. You can take a look at your blue bar example, though there is also an issue of wider gamut, Abney Effect is one of the main reasons the blue looks so "purple". And now as we changed the working primaries to Rec.2020, the Notorious Six skewing is now coupled with the descrepency between primaries, the effect will be now amplified. Now only Rec.2020's R, G, B will be attenuated chromaticity-linearly. And Rec.2020 primaries have different chromaticity angles compare with Rec.709: Therefore, following per-channel mechanism's Notorious Six behavior, Rec.709 blue will be skewed towards magenta, Green will be twowards yellow, etc.. Some of this slewings might be desireable, some might be not, so we need a handle of control. The solution we applied is that, along with the inset, we also do a rotation of primaries, for example, rotate the blue towards cyan a little bit, so that it doesn't go to magenta. How much R, G, B each should rotate is a matter of taste and testings on various EXRs. Therefore you would see different implentations of AgX tend to have their own fine tuned inset-and-rotation matrix. Again, the matrix used by Blender is included in the scripts I shared. Specifically from
This is the matrix implemented in Blender, to be used in Linear Rec.2020. Again, different implementations of AgX tend to use their own tested inset-and-rotation matrix, so this is just "my tested matrix", not "The Matrix". Feel free to generate your own matrix from the SB2383 script and fine tune it against as many challenging EXRs as possible. Apart from that, now we have moved the working primaries to Rec.2020, the image that AgX forms out of the box is now in Rec.2020 display hardware medium. It will contain values outside of Rec.709 display medium. Direct color matrix transform from Rec.2020 to Rec.709 will be equivalent to a hard clamping, which may not be desireable at cases. In this case, yes we used the lower guard rail for that.
The simple explaination is basically:
That's basically what Sakari did in the script that he shared. I ended up adding a few lines to deal with real-camera footages that have negative luminance error. But overall it still pretty much works the same way. For reference, we used the Lower Guard Rail pre-formation in Linear Rec.2020, and then post-Rec.2020-formation as we try to produce the images for other display hardware mediums. |
@EaryChow Thanks for your explanations! The latest implement is based on Filament's version which is in turn based on Blenders so the inset and outset matrices are the same as the ones you've referenced. I've discussed with @donmccurdy offline and I think the current state of the PR is good to merge. The implementation is currently based on Filament's real time implementation and the look mostly matches the Blender results aside from brightness. Unfortunately because Blender's "guard rail" approach to accommodate negative values requires analyzing the full image it's not something we can easily fit into the current three.js fragment-based tone mapping system - Filament doesn't support this either. I think we can leave it for future work potentially. To round things out here's the final set of image comparisons:
And screenshots from the tonemapping demo.
|
Fantastic work guys! Congratulations and thank you for all your hard work here 👏🏻 |
Do we still need a fallback for WebGL 1? |
Ah wait, now I see bbbd075 👍 . |
* Add support for AGX tone mapping * Update tonemapping example * Update test * AGX -> AgX * glsl lint, suffixes fix * Adjust tone mapping constant values * Update src/renderers/shaders/ShaderChunk/tonemapping_pars_fragment.glsl.js Co-authored-by: Don McCurdy <[email protected]> * Update src/renderers/shaders/ShaderChunk/tonemapping_pars_fragment.glsl.js Co-authored-by: Don McCurdy <[email protected]> * Update OutputPass.js Add `AgX` support. * Update OutputShader.js Add `AgX` support. * Clamp the values to be > 0.0 * Add rec2020 matrices * Fix rec2020 conversion matrices * Update implementation to be based on filament using rec 2020 color space * Spaces -> tabs * linting * Code cleanup * Support WebGL1, rearrange * Comments update * Remove redundant clamp --------- Co-authored-by: Don McCurdy <[email protected]> Co-authored-by: Michael Herzog <[email protected]>
Fix #27362
Related to #26479
Description
Adds support for AGX Tone Mapping based on the implementation from this article.
Do we know of any defacto test images for AGX to make sure it looks right?
Here are some comparison screenshots when env map intensity is set to 3.0: