Memory: Compress and Resize Textures

[mikhail-dcl]

New description

How it works now

• Textures are loaded by Unity Web Request
  • It decodes JPG and PNG only
  • Does not compress textures, PNG is loaded into RGBA32, JPG into RGB24
  • It is the only built-in way to download textures
• Without compression images have an enormous size, e.g. 2k RGBA takes 128 MB RAM
• We don't control all endpoints, users may expose images as they wish quickly bloating up memory usage. Currently 3rd party textures and textures hosted in Catalyst are TOP1 contributors to the memory usage

Expected Behavior

• Textures are decoded and compressed without intermediate results and allocations
• Texture compression is selected based on hardware capabilities, e.g. 2k ASTC 8x8 takes 8 MB
• It's possible to target a specified maximum textures resolution that a user can control from Settings

Nice to have

• Support for Webp/Gif
• Support for different compression quality to target different hardware tiers more aggressively

Possible Solution

• Integrate any Rust/C++/C# that can decode and compress a texture from the given byte pointer without additional managed allocations.
  • As a bonus majority of the existing plugins support more formats out of the box
  • There are very complex solutions that would require a lot of bridging, a weighed one should be selected according to our needs
• Change the current way of loading textures to loading a native byte array instead and passing it for decoding into the plugin
• Pre-allocate the texture on Unity side with the selected format
• The result should be GPU-compatible straight-away so it is possible to write it directly into the preallocated texture or via this

Benefits

• We will experience immediate memory relief, memory consumption will be more predictable for users
• We don't need to make a design decision to forbid all 3rd links to textures and move everything to our catalyst, we can implement it purely on our side within the predictable period of time

Old description

Output should be a texture in a GPU-friendly format (that can be uploaded to GPU without additional conversions, e.g. by SetRawData)

It's possible to retrieve textures from an external source; in this case it can't be preprocessed

Unity does not expose API for decoding textures on a worker thread. Unity support recommended using an external library, e.g. FreeImage

I tried to integrate it in the past without any success, and it was not straightforward at all.
I found a topic where a guy tried to do the same without success either.

Just like for Meshes we should introduce different levels of LODs, LODs can be used as mip-maps

[mikhail-dcl]

I have a plan in my head regarding textures:
Integrate a library for decoding (FreeImage)
Integrate a library for runtime compression (similar to https://docs.unity3d.com/ScriptReference/Texture2D.Compress.html) that can run on a worker thread (I'm talking to Unity to find one)
For textures web requests: use a preallocated buffer, just like we discussed before
On a background thread decode and compress
Take a texture from the pool (in case it is similarly sized we can set data to the internal array directly, otherwise reload the internal array)
This technique can be applied to both single textures and gifs

[mikhail-dcl]

Early research showed the following conclusions:

• I integrated A .NET wrapper of FreeImage with ease and was able to run the image decoding on the background thread and then write raw data back to Texture2D:

  • This wrapper has Microsoft dependencies that we don't need, we must clean it up before we can use it: consider forking off, providing an API for Texture2D, and then integrating.

  • Libraries for Win32/64, MacOS, and Linux were included. FreeImage itself provides build scripts for many platforms, it should be easy to rebuild in case of a need

  • Reading pixels looks like this:

    var texture = new Texture2D(width, height, format, false);
    
     // TODO add mip-maps with `Rescale`
     NativeArray<byte> rawBits = texture.GetPixelData<byte>(0);
     
     void* unmanagedPointer = rawBits.GetUnsafePtr();
     // GetBits() is a native function
     IntPtr unmanagedBits = GetBits(dib);
    
     Unsafe.CopyBlock(unmanagedPointer, unmanagedBits.ToPointer(), (uint)rawBits.Length);
    
     texture.Apply();
    

  • I could not find a way to support 8bit textures, I tried using GraphicsFormat but Unity asserts it (probably a Unity bug). This should not be a problem as we can store such textures as is, they occupy less memory than fully-colored ones

  • For some reason, the internal representation is BGRA (Blue and Red channels are swapped) so I had to swap them manually like this. it is happening via the native API on the background thread so can be considered as neglectable

     FIBITMAP redChannel = GetChannel(dib, FREE_IMAGE_COLOR_CHANNEL.FICC_RED);
     FIBITMAP blueChannel = GetChannel(dib, FREE_IMAGE_COLOR_CHANNEL.FICC_BLUE);
    
     SetChannel(dib, redChannel, FREE_IMAGE_COLOR_CHANNEL.FICC_BLUE);
     SetChannel(dib, blueChannel, FREE_IMAGE_COLOR_CHANNEL.FICC_RED);
    
     UnloadEx(ref redChannel);
     UnloadEx(ref blueChannel);
    

  • It is possible to rescale textures on a background thread and then upload them as mipmaps

  • It does allocate unmanaged memory that we are responsible to release, it's fine as it does not involve GC

• I tried to integrate Compressonator to compress into Desktop format at runtime
  • This project is super mature and feature-reach, supporting the most modern formats such BC7
  • It provides scripts for every standalone platform so it can be built with ease
  • I didn't find any wrappers for C#
  • So I started to write them on my own, but I didn't finish. I don't see any limitations why it should not work: it's a pretty standard flow: just write interop bindings, include native libraries for each platform, and do whatever you want. Then take a pointer to the natively allocated array of colors, and write it to the NativeArray of the Texture: it supports compressed formats on creation.

[pravusjif]

Kudos to you, based on your notes your approach looks promising!

You probably already saw this but just in case it's useful for you in any way, FYI for the WebGL client, the kernel processes GIFs in a webworker/thread, stores the texture and sends a pointer to the texture to unity and in unity the image is loaded from that pointer.

[NickKhalow]

@mikhail-dcl @m3taphysics do we have a test set of images that we would support? I need it to make a playground and to test the solution

[NickKhalow]

@mikhail-dcl FreeImage uses BGR by default on Little-Endian machines, we have to set it manually during lib's compilation

// Color-Order:
// The specified order of color components red, green and blue affects 24-
// and 32-bit images of type FIT_BITMAP as well as the colors that are part
// of a color palette. All other images always use RGB order. By default,
// color order is coupled to endianness:
// little-endian -> BGR
// big-endian -> RGB
// However, you can always define FREEIMAGE_COLORORDER to any of the known
// orders FREEIMAGE_COLORORDER_BGR (0) and FREEIMAGE_COLORORDER_RGB (1) to
// specify your preferred color order.
#define FREEIMAGE_COLORORDER_BGR 0
#define FREEIMAGE_COLORORDER_RGB 1
#if (!defined(FREEIMAGE_COLORORDER)) || ((FREEIMAGE_COLORORDER != FREEIMAGE_COLORORDER_BGR) && (FREEIMAGE_COLORORDER != FREEIMAGE_COLORORDER_RGB))
#if defined(FREEIMAGE_BIGENDIAN)
#define FREEIMAGE_COLORORDER FREEIMAGE_COLORORDER_RGB
#else
#define FREEIMAGE_COLORORDER FREEIMAGE_COLORORDER_BGR
#endif // FREEIMAGE_BIGENDIAN
#endif // FREEIMAGE_COLORORDER

[NickKhalow]

It's finished partly with #2568 PR, MipMap and Gif supports remain to do