Add `ImageSamplerDescriptor` as an image loader setting #9982

Kanabenki · 2023-10-01T07:39:07Z

Objective

Add a new type mirroring wgpu::SamplerDescriptor for ImageLoaderSettings to control how a loaded image should be sampled.

Fix issues with texture sampler descriptors not being set when loading gltf texture from URI.

Solution

Add a new ImageSamplerDescriptor and its affiliated types that mirrors wgpu::SamplerDescriptor, use it in the image loader settings.

Changelog

Added

Added new types ImageSamplerDescriptor, ImageAddressMode, ImageFilterMode, ImageCompareFunction and ImageSamplerBorderColor that mirrors the corresponding wgpu types.
ImageLoaderSettings now carries an ImageSamplerDescriptor field that will be used to determine how the loaded image is sampled, and will be serialized as part of the image assets .meta files.

Changed

Image::from_buffer now takes the sampler descriptor to use as an additional parameter.

Fixed

Sampler descriptors are set for gltf textures loaded from URI.

superdump · 2023-10-01T09:47:34Z

crates/bevy_render/Cargo.toml

@@ -57,7 +57,8 @@ bevy_tasks = { path = "../bevy_tasks", version = "0.12.0-dev" }
 image = { version = "0.24", default-features = false }

 # misc
-wgpu = { version = "0.16.0", features=["naga"] }
+wgpu = { version = "0.16.0", features = ["naga"] }
+wgpu-types = { version = "0.16.0", features = ["trace", "replay"] }


Why is wgpu-types needed?

I added it to enable the trace and replay features unconditionally only for the types re-exported in wgpu from there (all types used in the fields of wgpu::SamplerDescriptor), without enabling the trace/replay related stuff in wgpu itself. But I can alos move the enabled features to wgpu.

Do we need to enable these if we're using a proxy type? wgpu is already pretty slow to compile as is, so I'd prefer to avoid making it take even longer by enabling serde.

It is still needed since the proxy type is used for the Serialize/Deserialize impl of wgpu::SamplerDescriptor, but the Serialize/Deserialize impl of its fields is pulled from the serde derives in wgpu_types. This should only impact the compile time of wgpu_types which is just a collection of types re-exported in wgpu, from some quick testing the compile times of wgpu itself aren't affected, and wgpu_types goes from 1.2s to 1.7s to build when those features are enabled.

Fwiw if we're going with this solution I'd rather we go with declaring a dependency on wgpu-types and enable the serde features on it (trace+replay), vs enabling them for the whole wgpu crate. (wgpu always includes wgpu-types, so this isn't adding another dependency, it's just not slowing down our compiles as much as enabling trace+replay for wgpu would).

superdump · 2023-10-01T09:48:18Z

crates/bevy_render/src/texture/image.rs

+/// Proxy type used to implement [`Serialize`] and [`Deserialize`] for [`ImageSampler`].
+#[derive(Serialize, Deserialize)]
+#[serde(remote = "wgpu::SamplerDescriptor<'static>")]
+struct WgpuSamplerDescriptor {
+    #[serde(skip)]
+    label: wgpu::Label<'static>,
+    address_mode_u: wgpu::AddressMode,
+    address_mode_v: wgpu::AddressMode,
+    address_mode_w: wgpu::AddressMode,
+    mag_filter: wgpu::FilterMode,
+    min_filter: wgpu::FilterMode,
+    mipmap_filter: wgpu::FilterMode,
+    lod_min_clamp: f32,
+    lod_max_clamp: f32,
+    compare: Option<wgpu::CompareFunction>,
+    anisotropy_clamp: u16,
+    border_color: Option<wgpu::SamplerBorderColor>,
+}
+


Can we newtype the wgpu SamplerDescriptor type instead of reimplementing it?

From my understanding, that's not possible using serde derive, a manual Serialize/Deserialize impl would need to be added for ImageSampler. Is that preferable ?

Trying to think what would be most maintainable. I am concerned about desync.

i think this is fine - At compile time, Serde will check that all the fields in the definition you provided match the fields in the remote type

superdump

Left a couple of questions.

mockersf · 2023-10-01T21:55:52Z

this issue used to be solved before Asset V2 by loading images directly from their path inside the gltf loader instead of calling the image loader

Kanabenki · 2023-10-02T06:54:10Z

I see, if that is preferable I'll do it that way. However won't we lose the parallelization of the textures loading by not going through the ImageLoader ?

mockersf · 2023-10-09T23:35:44Z

We were doing parallelisation directly inside the loader: https://github.com/bevyengine/bevy/blob/release-0.11.3/crates/bevy_gltf/src/loader.rs#L383-L429

And the gltf texture loader was able to handle both path and data directly: https://github.com/bevyengine/bevy/blob/release-0.11.3/crates/bevy_gltf/src/loader.rs#L566-L616

cart · 2023-10-10T18:55:36Z

this issue used to be solved before Asset V2 by loading images directly from their path inside the gltf loader instead of calling the image loader

I think the "load" approach is preferable for a number of reasons:

It allows these images to be pre-processed (compressed / gpu-friendly format) cleanly and separately. It is possible with embedded images, just more complicated. "one to many processed assets files" will make the "embedded processing" story a bit cleaner.
It allows users to configure each image, if they want, by configuring the metadata files on each image.

We might want to make this configurable, but imo "separate image" is the correct default.

mockersf · 2023-10-10T20:36:44Z

I think the "load" approach is preferable

For me, the most important is to use the same path to load the data, wether it's inline or in a separate file. That difference introduced a lot of bugs in the past, and introduce new ones now

Elabajaba · 2023-10-11T04:29:15Z

Needs to be updated since we landed the wgpu update PR.

Kanabenki · 2023-10-11T10:21:37Z

What would be the best course of action for this PR while one to many asset loading is not supported ? For embedded textures, would using the label part of the AssetPath to distinguish those be ok ?

Otherwise I could revert to the pre-asset V2 way with everything in the gltf loader, or keep the split handling between embedded/external texture, and in both case handle everything through ImageLoader in a future PR once one to many asset loading is implemented.

cart · 2023-10-13T23:42:52Z

I think the approach in this PR is the right path. Loading images (and other assets) rather than embedding them is a pattern we will use for other scene types too (ex: Bevy Scenes). I still think it is the right path here.

Additionally, we want to add sampler descriptor configuration to the ImageLoader settings regardless of what we choose to do in this PR. When a developer defines a sprite png file, they will want to be able to define the filters on a per-asset basis. Configurable per-image-asset filters will likely be one of the highest-frequency use cases for "asset meta files".

I do share concerns about surfacing wgpu types directly here. Not only for the desync risk, but also because this will be serialized to meta files. We need to be in full control of that schema so we can detect and do migrations automatically across Bevy versions. The docs for these types should call out that changes to the schema will require migrations in some cases. (migrations have not yet been implemented ... this is a must-have feature for the next release cycle).

For this PR, I think we should create a new ImageSamplerDescriptor type. Regrettably, I think we will want to mirror the other types too (AddressMode, FilterMode, CompareFunction, SamplerBorderColor). These type names should also have an Image prefix. These types should all derive Serialize/Deserialize and we should manually define conversion to/from the wgpu types.

I think we should re-contextualize this PR to focus on configurable samplers in meta files / the title should be adjusted to accommodate that). Or alternatively, that should be broken into a separate PR and merged first. Imo defining samplers in meta files is a "big ticket" feature that should be the focus of this change set (from a user perspective).

Kanabenki · 2023-10-15T13:05:25Z

Thanks for the detailed answer. I'll submit a separate PR to add bevy-defined sampling settings and rework this one to use those, then.

One thing I'm now wondering is if it actually make sense for the sampling settings to be part of the ImageLoaderSettings ? Having different sampling configuration for the same image seems like a valid use case (and is possible in gltf since a texture combine an image with a sampler), which doesn't seem possible if the sampling configuration is directly tied to an asset path. Maybe that would be better fitted for a setting on each material that use the texture, or on an intermediate "texture" asset that would group an image with a sampling setting.

That's a bit tangential to the gltf texture loading since in that case we can pass different ImageLoaderSettings, although I suppose it could still be a problem if one were to use an AssetProcessor with compressed texture saving, since it could map several gltf textures with differents gltf samplers but the same image to an unique compressed image for which only one sampling setting would be saved.

Diddykonga · 2023-10-20T23:50:46Z

Want to mention, when helping people with Godot, one of the changes they made was to remove Filter Mode from Image Assets, as many people were upset that they had to duplicate their images, just to have different Filtering applied to them, they moved this to the Material/Drawer that used them, I believe.

robtfm · 2023-10-21T09:23:26Z

crates/bevy_gltf/src/loader.rs

                load_context.load_with_settings(path, move |settings: &mut ImageLoaderSettings| {
                    settings.is_srgb = is_srgb;
+                    settings.sampler_descriptor = sampler_descriptor.clone();


Suggested change

settings.sampler_descriptor = sampler_descriptor.clone();

settings.sampler_descriptor = sampler_descriptor;

robtfm · 2023-10-21T09:24:08Z

crates/bevy_gltf/src/loader.rs

@@ -644,6 +649,7 @@ async fn load_image<'a, 'b>(
    supported_compressed_formats: CompressedImageFormats,
 ) -> Result<ImageOrPath, GltfError> {
    let is_srgb = !linear_textures.contains(&gltf_texture.index());
+    let sampler_descriptor = ImageSampler::Descriptor(texture_sampler(&gltf_texture));


please use this descriptor on line 664 instead of creating a new one

robtfm · 2023-10-21T09:27:15Z

crates/bevy_gltf/src/loader.rs

+                    supported_compressed_formats,
+                    is_srgb,
+                )?;
+                image.sampler_descriptor = sampler_descriptor;


the sampler should probably be an argument to Image::from_buffer - it is manually set after calling in 3 of 4 callsites (and it's easy to construct the default variant in the remaining case).

robtfm · 2023-10-21T09:34:49Z

crates/bevy_render/src/texture/image.rs

+/// Proxy type used to implement [`Serialize`] and [`Deserialize`] for [`ImageSampler`].
+#[derive(Serialize, Deserialize)]
+#[serde(remote = "wgpu::SamplerDescriptor<'static>")]
+struct WgpuSamplerDescriptor {
+    #[serde(skip)]
+    label: wgpu::Label<'static>,
+    address_mode_u: wgpu::AddressMode,
+    address_mode_v: wgpu::AddressMode,
+    address_mode_w: wgpu::AddressMode,
+    mag_filter: wgpu::FilterMode,
+    min_filter: wgpu::FilterMode,
+    mipmap_filter: wgpu::FilterMode,
+    lod_min_clamp: f32,
+    lod_max_clamp: f32,
+    compare: Option<wgpu::CompareFunction>,
+    anisotropy_clamp: u16,
+    border_color: Option<wgpu::SamplerBorderColor>,
+}
+


i think this is fine - At compile time, Serde will check that all the fields in the definition you provided match the fields in the remote type

Kanabenki · 2023-10-21T13:23:55Z

I've updated this PR with the discussed changes instead of doing a separate one, since that one would have ended up with mostly nothing in it.

Some outstanding questions:

As discussed in the last comments, are we sure we want to have all sampling-related settings on the ImageLoaderSettings instead of allowing different sampling settings for an image, whether that be at the material level or through an additional asset type (or anything else)?
I didn't add a wgpu::SamplerDescriptor -> ImageSamplerDescriptor conversion since there's no use for it in the bevy codebase currently, do we still want to provide it for the users? (edit: nevermind, it is needed for the compressed texture saver)
I choose to mirror the wgpu types exactly, that includes some wgpu feature gated settings like ImageSamplerBorderColor. Is that ok or should those be removed?

…rsion

Kanabenki · 2023-10-21T13:50:03Z

Want to mention, when helping people with Godot, one of the changes they made was to remove Filter Mode from Image Assets, as many people were upset that they had to duplicate their images, just to have different Filtering applied to them, they moved this to the Material/Drawer that used them, I believe.

Indeed, I discussed that on Discord but forgot to add that info here! From what I saw most mainstream engines still put all sampling-related settings at the image import level, except for Godot which made the switch for 4.0.

Kanabenki · 2023-10-21T13:52:17Z

crates/bevy_render/src/texture/compressed_image_saver.rs

@@ -30,6 +33,10 @@ impl AssetSaver for CompressedImageSaver {
        compressor_params.set_basis_format(basis_universal::BasisTextureFormat::UASTC4x4);
        compressor_params.set_generate_mipmaps(true);
        let is_srgb = image.texture_descriptor.format.is_srgb();
+        let sampler_descriptor = match &image.sampler_descriptor {


Not too sure about that one, if we want to use the actual default sampler from the render plugin, it would require either passing its descriptor here or changing ImageLoaderSettings::sampler_descriptor to an ImageSampler and using ImageSamplerDescriptor instead of wgpu::SamplerDescriptor in ImageSampler.

cart · 2023-10-25T00:48:00Z

As discussed in the last comments, are we sure we want to have all sampling-related settings on the ImageLoaderSettings instead of allowing different sampling settings for an image, whether that be at the material level or through an additional asset type (or anything else)?

For now: I think the answer is yes. Moving it to the material level will require more time for discussion than we have for 0.12. The current approach is consistent with the current Image / Texture model.

I do think its worth having that conversation, just not here/now.

I choose to mirror the wgpu types exactly, that includes some wgpu feature gated settings like ImageSamplerBorderColor. Is that ok or should those be removed?

I think thats the right move. We can just ignore the settings when they aren't supported.

cart · 2023-10-25T00:56:19Z

I have a number of changes queued up that accomplish a couple of goals:

Treat ImageSamplerDescriptor as the "public descriptor api" in all places (rather than leaking the wgpu internals)
Make it possible to use the "runtime configured default" sampler in meta files. (this addresses your open comment)

I can't push those changes as it looks like you've restricted the ability for maintainers to push changes to your branch. I think I might just merge this as-is and then create a new PR for my changes.

cart · 2023-10-25T01:38:33Z

Also found a bug that was preventing the configured samplers from being used for Basis, KTX2, and DDS textures. Fixed that on my branch.

cart · 2023-10-25T01:49:36Z

PR is here: #10254

cart

This is the right path. I do have a few changes queued up, which we can review and merge after this.

Kanabenki · 2023-10-25T06:45:49Z

I can't push those changes as it looks like you've restricted the ability for maintainers to push changes to your branch. I think I might just merge this as-is and then create a new PR for my changes.

I don't remember changing that if that isn't the default, I'll make sure to have it enabled for my next PRs 🙂

# Objective - Build on the changes in #9982 - Use `ImageSamplerDescriptor` as the "public image sampler descriptor" interface in all places (for consistency) - Make it possible to configure textures to use the "default" sampler (as configured in the `DefaultImageSampler` resource) - Fix a bug introduced in #9982 that prevents configured samplers from being used in Basis, KTX2, and DDS textures --- ## Migration Guide - When using the `Image` API, use `ImageSamplerDescriptor` instead of `wgpu::SamplerDescriptor` - If writing custom wgpu renderer features that work with `Image`, call `&image_sampler.as_wgpu()` to convert to a wgpu descriptor.

) Closes bevyengine#9946 # Objective Add a new type mirroring `wgpu::SamplerDescriptor` for `ImageLoaderSettings` to control how a loaded image should be sampled. Fix issues with texture sampler descriptors not being set when loading gltf texture from URI. ## Solution Add a new `ImageSamplerDescriptor` and its affiliated types that mirrors `wgpu::SamplerDescriptor`, use it in the image loader settings. --- ## Changelog ### Added - Added new types `ImageSamplerDescriptor`, `ImageAddressMode`, `ImageFilterMode`, `ImageCompareFunction` and `ImageSamplerBorderColor` that mirrors the corresponding wgpu types. - `ImageLoaderSettings` now carries an `ImageSamplerDescriptor` field that will be used to determine how the loaded image is sampled, and will be serialized as part of the image assets `.meta` files. ### Changed - `Image::from_buffer` now takes the sampler descriptor to use as an additional parameter. ### Fixed - Sampler descriptors are set for gltf textures loaded from URI.

# Objective - Build on the changes in bevyengine#9982 - Use `ImageSamplerDescriptor` as the "public image sampler descriptor" interface in all places (for consistency) - Make it possible to configure textures to use the "default" sampler (as configured in the `DefaultImageSampler` resource) - Fix a bug introduced in bevyengine#9982 that prevents configured samplers from being used in Basis, KTX2, and DDS textures --- ## Migration Guide - When using the `Image` API, use `ImageSamplerDescriptor` instead of `wgpu::SamplerDescriptor` - If writing custom wgpu renderer features that work with `Image`, call `&image_sampler.as_wgpu()` to convert to a wgpu descriptor.

) Closes bevyengine#9946 # Objective Add a new type mirroring `wgpu::SamplerDescriptor` for `ImageLoaderSettings` to control how a loaded image should be sampled. Fix issues with texture sampler descriptors not being set when loading gltf texture from URI. ## Solution Add a new `ImageSamplerDescriptor` and its affiliated types that mirrors `wgpu::SamplerDescriptor`, use it in the image loader settings. --- ## Changelog ### Added - Added new types `ImageSamplerDescriptor`, `ImageAddressMode`, `ImageFilterMode`, `ImageCompareFunction` and `ImageSamplerBorderColor` that mirrors the corresponding wgpu types. - `ImageLoaderSettings` now carries an `ImageSamplerDescriptor` field that will be used to determine how the loaded image is sampled, and will be serialized as part of the image assets `.meta` files. ### Changed - `Image::from_buffer` now takes the sampler descriptor to use as an additional parameter. ### Fixed - Sampler descriptors are set for gltf textures loaded from URI.

# Objective - Build on the changes in bevyengine#9982 - Use `ImageSamplerDescriptor` as the "public image sampler descriptor" interface in all places (for consistency) - Make it possible to configure textures to use the "default" sampler (as configured in the `DefaultImageSampler` resource) - Fix a bug introduced in bevyengine#9982 that prevents configured samplers from being used in Basis, KTX2, and DDS textures --- ## Migration Guide - When using the `Image` API, use `ImageSamplerDescriptor` instead of `wgpu::SamplerDescriptor` - If writing custom wgpu renderer features that work with `Image`, call `&image_sampler.as_wgpu()` to convert to a wgpu descriptor.

# Objective - Build on the changes in bevyengine/bevy#9982 - Use `ImageSamplerDescriptor` as the "public image sampler descriptor" interface in all places (for consistency) - Make it possible to configure textures to use the "default" sampler (as configured in the `DefaultImageSampler` resource) - Fix a bug introduced in #9982 that prevents configured samplers from being used in Basis, KTX2, and DDS textures --- ## Migration Guide - When using the `Image` API, use `ImageSamplerDescriptor` instead of `wgpu::SamplerDescriptor` - If writing custom wgpu renderer features that work with `Image`, call `&image_sampler.as_wgpu()` to convert to a wgpu descriptor.

Kanabenki force-pushed the gtlf-sampler-descriptor branch from e9ce9ea to a60d87c Compare October 1, 2023 07:48

alice-i-cecile added C-Bug An unexpected or incorrect behavior A-Rendering Drawing game state to the screen A-Assets Load files from disk to use for things like images, models, and sounds labels Oct 1, 2023

superdump reviewed Oct 1, 2023

View reviewed changes

cart added this to the 0.12 milestone Oct 10, 2023

Kanabenki force-pushed the gtlf-sampler-descriptor branch from a60d87c to 42f1d63 Compare October 11, 2023 10:03

robtfm reviewed Oct 21, 2023

View reviewed changes

Kanabenki added 2 commits October 21, 2023 12:30

Define ImageSamplerDescriptor that mirrors wgpu::SamplerDescriptor

200a44f

Use ImageSamplerDescriptor in ImageLoaderSettings

53387be

Kanabenki force-pushed the gtlf-sampler-descriptor branch from 42f1d63 to 53387be Compare October 21, 2023 13:23

Kanabenki changed the title ~~Set sampler descriptors for gltf image URI~~ Add ImageSamplerDescriptor as an image loader setting Oct 21, 2023

Implement wgpu::SamplerDescriptor to ImageSamplerDescriptor conve…

7cd7b52

…rsion

Kanabenki commented Oct 21, 2023

View reviewed changes

cart mentioned this pull request Oct 25, 2023

Image Sampler Improvements #10254

Merged

cart approved these changes Oct 25, 2023

View reviewed changes

cart added this pull request to the merge queue Oct 25, 2023

Merged via the queue into bevyengine:main with commit 756fb06 Oct 25, 2023
21 checks passed

cart mentioned this pull request Oct 30, 2023

News: Release 0.12 bevyengine/bevy-website#754

Merged

43 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `ImageSamplerDescriptor` as an image loader setting #9982

Add `ImageSamplerDescriptor` as an image loader setting #9982

Kanabenki commented Oct 1, 2023 •

edited

Loading

superdump Oct 1, 2023

Kanabenki Oct 1, 2023

Elabajaba Oct 1, 2023

Kanabenki Oct 2, 2023

Elabajaba Oct 11, 2023

superdump Oct 1, 2023

Kanabenki Oct 1, 2023

superdump Oct 13, 2023

robtfm Oct 21, 2023

superdump left a comment

mockersf commented Oct 1, 2023

Kanabenki commented Oct 2, 2023 •

edited

Loading

mockersf commented Oct 9, 2023 •

edited

Loading

cart commented Oct 10, 2023

mockersf commented Oct 10, 2023

Elabajaba commented Oct 11, 2023

Kanabenki commented Oct 11, 2023

cart commented Oct 13, 2023 •

edited

Loading

Kanabenki commented Oct 15, 2023 •

edited

Loading

Diddykonga commented Oct 20, 2023

robtfm Oct 21, 2023

robtfm Oct 21, 2023

robtfm Oct 21, 2023

robtfm Oct 21, 2023

Kanabenki commented Oct 21, 2023 •

edited

Loading

Kanabenki commented Oct 21, 2023

Kanabenki Oct 21, 2023

cart commented Oct 25, 2023

cart commented Oct 25, 2023 •

edited

Loading

cart commented Oct 25, 2023 •

edited

Loading

cart commented Oct 25, 2023

cart left a comment

Kanabenki commented Oct 25, 2023

	settings.sampler_descriptor = sampler_descriptor.clone();
	settings.sampler_descriptor = sampler_descriptor;

Add ImageSamplerDescriptor as an image loader setting #9982

Add ImageSamplerDescriptor as an image loader setting #9982

Conversation

Kanabenki commented Oct 1, 2023 • edited Loading

Objective

Solution

Changelog

Added

Changed

Fixed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

superdump left a comment

Choose a reason for hiding this comment

mockersf commented Oct 1, 2023

Kanabenki commented Oct 2, 2023 • edited Loading

mockersf commented Oct 9, 2023 • edited Loading

cart commented Oct 10, 2023

mockersf commented Oct 10, 2023

Elabajaba commented Oct 11, 2023

Kanabenki commented Oct 11, 2023

cart commented Oct 13, 2023 • edited Loading

Kanabenki commented Oct 15, 2023 • edited Loading

Diddykonga commented Oct 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kanabenki commented Oct 21, 2023 • edited Loading

Kanabenki commented Oct 21, 2023

Choose a reason for hiding this comment

cart commented Oct 25, 2023

cart commented Oct 25, 2023 • edited Loading

cart commented Oct 25, 2023 • edited Loading

cart commented Oct 25, 2023

cart left a comment

Choose a reason for hiding this comment

Kanabenki commented Oct 25, 2023

Add `ImageSamplerDescriptor` as an image loader setting #9982

Add `ImageSamplerDescriptor` as an image loader setting #9982

Kanabenki commented Oct 1, 2023 •

edited

Loading

Kanabenki commented Oct 2, 2023 •

edited

Loading

mockersf commented Oct 9, 2023 •

edited

Loading

cart commented Oct 13, 2023 •

edited

Loading

Kanabenki commented Oct 15, 2023 •

edited

Loading

Kanabenki commented Oct 21, 2023 •

edited

Loading

cart commented Oct 25, 2023 •

edited

Loading

cart commented Oct 25, 2023 •

edited

Loading