Consider to resize textures for low-end devices

[takahirox]

Is your feature request related to a problem? Please describe.

We uses MeshBasicMaterial based material if the material quality mode is low. But we reuse the original textures. If the original textures are huge, they can be slow to be rendered/fetched or consume memory too much on devices on which users want to use low quality mode.

Describe the solution you'd like

Consider to resize huge texture image sizes on the fly, for example to 512x512.

Note: Textures can't be resized if compressed textures?

Describe alternatives you've considered

Introduce LOD support. In low quality mode, we use only low levels. If the assets are created properly, probably low levels should use smaller textures.

[takahirox]

I want to move this forward because

1. it can fix [iOS] The page is refreshed when uploading certain objects from Sketchfab #4669
2. (I will need to test well enough to confirm but my current impression is) the main thread blocking time of image resize + smaller texture upload can be shorter than the one of huge texture upload
3. memory friendly especially on limited VRAM devices
4. Even after introducing LOD, texture resize is still useful

Additional context:

We may be able to think of offloading the resize from the main thread by using OffscreenCanvas) (and/or createImageBitmap?) although I'm not sure if it's doable on all major platforms.

[takahirox]

I'm thinking how to expose the configuration. Some ideas.

1. Force to resize the textures on certain platforms. For example force to resize textures up to 512x512 on mobile devices
2. Bind to the material quality mode preference. For example resize textures up to 2048x2048 with medium mode and to 512x512 with low mode
3. Add a new preference "Maximum texture size"

I prefer 3, and the default value can be set on mobile devices, for example 512x512.

[brianpeiris]

Option 3 with a default of 512x512 for mobile sounds like a good idea to me.

Some other things to consider:

• Should we do texture resizing for the entire scene, or only for objects in the scene?
• Should we avoid resizing textures on some objects? Like Image media, or PDFs? Maybe texture resizing will make some scenes look really bad?
• Should we do this for all media sources, or just ones that we know produce heavy objects (Sketchfab)
• Sketchfab already has an off-thread worker for unzipping the model. Should we do texture resizing inside that worker?
  • https://github.com/mozilla/hubs/blob/master/src/components/gltf-model-plus.js#L59
  • https://github.com/mozilla/hubs/blob/master/src/workers/sketchfab-zip.worker.js
• Should we avoid doing this on tablet devices that may have more VRAM?
• In the future, should we consider doing texture resizing as a backend service, so that it doesn't need to be done on each client device?

None of those are blockers though. I think we can benefit from this even if we do the simplest possible implementation, and then iterate on it as needed afterwards.

[keianhzo]

I agree with all the comments above, just a couple of questions:

• How would this affect the loading times for big scenes with a lot of textures? I guess loading times can grow exponentially as we have to wait for all textures to be compressed in sequence or small batches?
• Can we cache the resized textures so we can skip the resizing step in future loads?
• Maybe adding some hint in Blender/Spoke so the artist can skip this for certain elements?

I also agree that server side assets processing would definitely be the ideal solution as:

• We could provide different assets bundles for different devices and benefit from the browser caching capabilities
• We can also do some other things as part of the process like texture compression
• Maybe we can also add this processing as part of the assets proxying so we can also do this for external content.

[takahirox]

Thanks for the comments.

• Should we do texture resizing for the entire scene, or only for objects in the scene?

I think of the both.

• Should we avoid resizing textures on some objects? Like Image media, or PDFs? Maybe texture resizing will make some scenes look really bad?

Good point. We need to test to check if medias are visible enough even with smaller textures. Ideally I want to resize all the textures.

• Should we do this for all media sources, or just ones that we know produce heavy objects (Sketchfab)

I'm thinking of all.

• Sketchfab already has an off-thread worker for unzipping the model. Should we do texture resizing inside that worker?

Ideally yes. Future work for me.

• Should we avoid doing this on tablet devices that may have more VRAM?

Maybe. Or we may relax for tablet devices, for example 512x512 for mobile devices and 1024x1024 for tablet devices.

• In the future, should we consider doing texture resizing as a backend service, so that it doesn't need to be done on each client device?
• Can we cache the resized textures so we can skip the resizing step in future loads?
• Maybe we can also add this processing as part of the assets proxying so we can also do this for external content.

Yes, those should be ones of our future options. But at first I'm thinking of just resizing in the main thread unless significant problems.

• How would this affect the loading times for big scenes with a lot of textures? I guess loading times can grow exponentially as we have to wait for all textures to be compressed in sequence or small batches?

Current my impression is, the main thread blocking time caused by texture resize + smaller texture upload is shorter than the one caused by huge texture upload. So we may expect loading time can be shorter. But a concern is, if we resize in the main thread, multiple textures resize will be done sequentially. Multiple (huge) textures upload can be done in parallel in the GPU thread? If so, the blocking time can be longer. Using workers may resolve. We may need to test.

• Maybe adding some hint in Blender/Spoke so the artist can skip this for certain elements?

Hm, currently I think all the textures should be resized. I may rethink if any practical use cases.

• We could provide different assets bundles for different devices and benefit from the browser caching capabilities

As I mentioned at another place, I'm thinking of using LOD (only using the lowest level of details for mobiles). And we may be able to create lower of details for mobiles on the fly somewhere, backend, server, or elsewhere.

• We can also do some other things as part of the process like texture compression

Maybe. One thing I remember is, when I tried texture compression with WASM on the fly it took a long time like a few minutes. At least texture compression should be done backend.

[takahirox]

In short, I want to start with resizing textures with Canvas (or createImageBitmap()) in the main thread and think of further optimizations later.

Because I want to quickly implement texture resizing for a serious iOS problem #4669 The further optimizations are interesting but may add some complexities to our software design/architecture. They should be discussed well.

[rawnsley]

This would be a great addition to Hubs and I would suggest broadening the scope beyond mobile. Some sort of LOD analysis would allow you to load textures up to an appropriate resolution and even to vary this dynamically, which could open the possibility of accommodating much larger scenes. It would also dovetail nicely with a progressive loading approach that could speed up room entry.

Regarding image resizing: for compressed textures it is strongly encouraged to include all the mipmap levels, and it's the default behaviour for the BasisU tool. This means you will usually have lower resolution versions available and could achieve everything you want by customising the mipmap loading code.

The most difficult part of all this is working out what threshold to apply to resolution. Small scenes will be fine at high res, but large scenes may still crash even with a low res cap. Also, the browser gives you very little insight into the amount of memory used or available, so the only real option is to set a budget heuristically.

[takahirox]

I changed my mind. I started to feel like starting with the option 2.

2. Bind to the material quality mode preference. For example resize textures up to 2048x2048 with medium mode and to 512x512 with low mode

Because

• What I want to quickly fix is [iOS] The page is refreshed when uploading certain objects from Sketchfab #4669. The option 2 would be good enough to resolve the problem.
• It doesn't add a big complexity
• The implementation would be easy. Just add texture resize logic on HubsMeshBasic(/Phong)Material creation

And I may start to with resizing textures only for glTF. I want to think of media (and other) textures later if needed.

I want to start to think of further dynamic texture resizing optimizations when we start to think of futher optimizations (Asset proxy server, dynamic LOD creation, Caching, and so on).

[takahirox]

Closing this issue.

I had two main purposes with the texture resize.

1. Resolve the crash on iOS. Huge textures seem to cause the crash on iOS. Texture resize may mitigete the problem.
2. Performance and memory optimization for low-end devices.

For 1, #5437 has been merged.

For 2, I realized that the visual quality of resized texture is much worse than my expectation. I want to revisit. Maybe optimization on asset creation with LOD and/or compressed textures. They don't have an effect to Sketchfab assets. I want to think of them separately.