How to persist data (textures) between renders

[Elias-Graf]

I'm trying to implement a Widget that renders a certain chess piece. Eventually it should support all of the pieces, but keeping it simple for this.

Currently I'm using this code, which I've seen a bunch:

struct ChessPiece {
    texture: Option<TextureHandle>,
    x: usize,
    y: usize,
}

impl ChessPiece {
    pub fn new(x: usize, y: usize) -> Self {
        Self {
            texture: None,
            x,
            y,
        }
    }
}

impl Widget for ChessPiece {
    fn ui(mut self, ui: &mut egui::Ui) -> egui::Response {
        const SIZE: egui::Vec2 = egui::vec2(50.0, 50.0);

        let pos = egui::pos2(self.x as f32, self.y as f32).to_vec2().mul(SIZE).to_pos2();
        let rect: egui::Rect = egui::Rect::from_min_max(pos, pos.add(SIZE));

        let texture = self
            .texture
            .get_or_insert(egui::Context::default().load_texture(
                "bishop_black",
                egui_extras::image::load_svg_bytes(svg_bytes_bishop_black).unwrap(),
            ));

        egui::Image::new(texture, SIZE).paint_at(ui, rect);

        ui.allocate_rect(rect, egui::Sense::click())
    }
}

Now, either I don't know how to correctly add widget to the UI, or I don't know how to correctly persist my texture.

I need to have that widget for every square. Thus I use the following code:

for y in 0..chess::Board::HEIGHT {
    for x in 0..chess::Board::WIDTH {
        ui.add(ChessPiece::new(x, y));
    }
}

That currently means the struct is re-created every render, which totally freezes the app. Any suggestions?

Note: The position calcu

[emilk]

Load the texture once, and reuse it. See https://github.com/emilk/egui/blob/0.17.0/eframe/examples/image.rs

You don't need to implement egui::Widget. You can just have a method which takes the texture by reference, and call it like paint_chess_piece(ui, [x, y], my_texture), for instance.

[Elias-Graf]

Yes, but I will need to load it at some point. Does this mean I need to load everything in the app itself? If yes, sounds kind of messy to me. I would have liked it to be abstracted away.

[coderedart]

I just realized, but in

        let texture = self
            .texture
            .get_or_insert(egui::Context::default().load_texture(
                "bishop_black",
                egui_extras::image::load_svg_bytes(svg_bytes_bishop_black).unwrap(),
            ));

you are making a new egui Context everytime you render a widget. I think you should be getting ctx with ui.ctx() instead. idk if it will lead to deadlock or not though.

[coderedart]

rather than creating a temporary widget. just use a BTreeMap with the row and column as key.
if you have a chess piece on all squares BTreeMap<(u8, u8), ChessPiece>
if you have some empty squares BTreeMap<(u8, u8), Option<ChessPiece>>

for ((x, y), piece) in board_map.iter() {
   ui.add(piece);
}

The only problem is texture duplication between various pieces if you use more than one ChessPiece referring to the same texture.
to avoid that, you can use an enum and store textures in a hashmap with enum as the key. when you want a piece, you use the texture if its already Some, otherwise, you load it.
but now, you cannot unload the textures that are not in use. so, you need to use something like texture manager that egui itself uses under the hood. each texture handle shares Arc<Mutex<HashMap>> which contains all textures and maintains a count of handles. as texture handles are dropped, they lock the map and reduce the count, and as they are cloned / created, they increase their count. when the final handle is dropped, it checks if the count is zero (no other texture handles alive), it deletes that texture from the map.

to avoid this complexity, if its just a simple chessboard. preload the textures in advance and keep them in a struct or enum_map when you first create the app itself.

[Elias-Graf]

Map

I've tried doing something like that, but ran into the exact same issue: Where do I store the map? I mean now we're one level up. Let's call the widget Board. How does it's memory persist?

In addition, this would be quite tedious, because the position data etc is stored in a separate struct, and I would have to remap that all the time. It would be much more convenient to just access the texture. I just want to use the library for the UI, not the logic.

TextureManager

I've actually tried using that API, I couldn't figure it out though. It would be really cool if there was some more user friendly texture manager, for example texture_manager.load("assets/bishop_black.png") which would then either create the texture, or return the cached one. Could just use the resource path as key.

Preloading the textures

Okay sure, it's just a simple app. I would still like to somehow keep them tydie in their widget but this seems to be a limitation of this library.

Then cache them globally. How would I go about constructing this? It would have to be a static variable, but the necessary APIs are not available in a static context, and also not marked as const.

[emilk]

You should be able to make a texture manager that uses egui_extras::RetainedImage

[Elias-Graf]

Okay, I'll go and try (again) to achieve that :). I would just really like to know how a widget can retain some state? I mean I don't need what you call a widget, I just want a component (similar to how web-fronted frameworks do it) to organize my code.

[emilk]

Well, textures/images should really not be part of the widget state, and having them managed by a global texture manager makes sense to me (if you write a good one, maybe make a PR to add it to egui_extras!).

Superficial widget state, like "Is this CollpsingHeader open or not?" is stored in Context::data.

Other state is your application state. For instance "where are the chess pieces?". That is up to you to store.

[coderedart]

https://docs.rs/egui/latest/egui/util/id_type_map/struct.IdTypeMap.html
I think, you can also get away by simply doing

let texture = self
            .texture
            .get_or_insert( // if self.text is None, get texture handle from `ui.data()` . this is to cache texture inside the widget struct.
               ui.data().get_temp_mut_or_insert_with( 
                 Id::new("bishop_black"),
                 | | 
                 // if `ui.data()` does not have it, then go get it from filesystem. 
                 // application wide caching of egui, so that this texture is loaded only once across the enter egui.
                 // load the texture from filesystem, and insert into `ui.data()`
                   egui_extras::image::load_svg_bytes(svg_bytes_bishop_black).unwrap()
            ).unwrap());

I haven't tried it, but if it works, its probably the easiest way to implement caching using pure egui.

[Elias-Graf]

@emilk So hey, I'm back. First up, take all of this with a grain of salt, I'm not all that experienced in rust, and although I'm interested in every field, most of my time I spent is developing in languages like TypeScript. And now the "but" lol, I may criticize how the textures are currently handled ^^ :).

First of all, this is the API I coded up:

API

const PIECE_PATH: &str = "src/assets/bishop_black.svg";
const PIECE_SIZE: texture_manager::Size = (255, 255);
const PIECE_SIZE_VEC: egui::Vec2 = egui::vec2(PIECE_SIZE.0 as f32, PIECE_SIZE.1 as f32);

// Obviously would be nicer if it were `ui.texture_manager`, but I didn't bother
// modifying the source code. More about this later in "integration" anyway.
let texture_manager = self.texture_manager.get_or_insert_with(|| {
    let mut tm = TextureManager::new(ui.ctx().tex_manager());

    // User-extensible parsing of files
    tm.set_byte_parser_for("svg", SVG_BYTES_PARSER);
    tm.set_byte_parser_for("png", png_bytes_parser);

    tm
});

// let texture_id = texture_manager.load(PIECE_PATH);
let texture_id = texture_manager.load_sized(PIECE_PATH, &PIECE_SIZE);

ui.image(texture_id, PIECE_SIZE_VEC)

User-extensible bytes parser / Flawed SVG parser

Given that a user might want to use all sorts of files, I thought I'd make the parsing extensible, especially because I've found the current SVG parser to be a bit flawed. To be specific, it always used the size specified in the SVG element, which meant, the image would never be able to be reasonable shown bigger than it was defined, which sort of is the point of SVGs as far as I'm aware.

Anyway, here is my implementation lol (you might find a lot of similarity with the one from egui_extras):

const SVG_BYTES_PARSER: texture_manager::BytesParser = |bytes, size| {
    let options = usvg::Options::default();
    let tree = usvg::Tree::from_data(&bytes, &options.to_ref()).unwrap();

    let (width, height) = match size {
        Some(size) => *size,
        None => {
            let svg_node_size = tree.svg_node().size;

            (
                svg_node_size.width() as usize,
                svg_node_size.height() as usize,
            )
        }
    };

    let mut pixmap = tiny_skia::Pixmap::new(width as u32, height as u32).unwrap();

    resvg::render(
        &tree,
        usvg::FitTo::Size(width as u32, height as u32),
        tiny_skia::Transform::default(),
        pixmap.as_mut(),
    )
    .unwrap();

    egui::ColorImage::from_rgba_unmultiplied([width as _, height as _], pixmap.data())
};

And here the png parser, which is basically a wrapper for load_image_bytes, and doesn't even work. It throws errors about not supporting "png"s, and I haven't bothered to look into that.

fn png_bytes_parser(bytes: Arc<[u8]>, _: Option<&texture_manager::Size>) -> egui::ColorImage {
    egui_extras::image::load_image_bytes(&bytes).unwrap()
}

Integration

In the current state, the API is very "external" from egui, basically wrapping the original texture manager (big part of my criticism to come). In the future, (from a user standpoint) it would be nice if it could be accessed like ctx.texture_manager.load(). In addition, it could somehow "hook" into the panting of the UI, and remove textures that haven't been utilized for a certain amount of render passes.

Existing texture API criticism

Again, take all of this with a grain of salt, I respect your work and don't mean to be rude.

First I implemented my texture manager so would cache ColorImages, or some underlying form of it. But then I noticed, that the ui.image() method, requires a texture id to be painted.
I thought to myself that that was an odd choice, and found out, that there is no (easy) way to paint an image without utilizing the existing texture manager. The unterlying Mesh depends on it.

So I've been asking myself, is there any reason to use those IDs and TextureHandles over a simple reference to the actual data? E.g. Rc<ColorImage> or Arc<ColorImage>. Currently I find the system to be quite brittle and inflexible. The way I see it (mind that my view on the project is quite limited), textures can only be drawn if they were allocated in the texture manager, in the same context as the image widget, making it basically impossible to implement a custom manager. Also those components are basically "hard-linked", making it difficult to re-purpose them. Another pitfall I encountered was trying to do egui::Context::default().load_texture(), keeping it separate from any instances, but I think that's not how that works.

So currently you can't have a mesh with a texture (which means no image?), without instantiating and maintaining the texture manager and co.

Implementation

So here is the implementation. It's probably not optimal, missing a few lines of documentation, and things like error handing are missing.

It was fun to implement and I learnt a lot, but I don't really fancy spending a bunch of extra time if it's not going to be utilized. The current version of it is definitely able to sustain my needs.

Feel free to get back to me though, if you want me to do some changes or have suggestions.

use eframe::{
    egui,
    epaint::{ColorImage, TextureId},
};

// TODO: Error handling

use std::{collections::HashMap, sync::Arc};

pub type Path = String;
pub type Size = (usize, usize);

/// Parses given bytes to a color image.
///
/// * `Arc<[u8]>` - bytes to be parsed
/// * [`Option<&Size>`] - the target for the processing - useful when rasterizing
pub type BytesParser = fn(Arc<[u8]>, Option<&Size>) -> ColorImage;

pub struct TextureManager {
    egui_texture_manager: Arc<eframe::egui::mutex::RwLock<eframe::epaint::TextureManager>>,

    bytes_cache: HashMap<Path, Arc<[u8]>>,
    texture_id_cache: HashMap<(Path, Size), TextureId>,

    bytes_parsers: HashMap<String, BytesParser>,
}

impl TextureManager {
    pub fn new(
        egui_texture_manager: Arc<eframe::egui::mutex::RwLock<eframe::epaint::TextureManager>>,
    ) -> Self {
        Self {
            egui_texture_manager,

            bytes_cache: HashMap::new(),
            texture_id_cache: HashMap::new(),

            bytes_parsers: HashMap::new(),
        }
    }

    /// Add a new parser for a given file (extension).
    pub fn set_byte_parser_for(&mut self, extension: &str, parser: BytesParser) {
        self.bytes_parsers.insert(extension.to_owned(), parser);
    }

    /// Load an image **without** specifying a size. Also see [`Self::load_sized()`].
    ///
    /// **Given that a size is needed for caching, size (0, 0) will be used.**
    ///
    /// The parser will not receive any size, and will be asked to infer it from
    /// the files content.
    pub fn load(&mut self, path: &str) -> TextureId {
        self.internal_load(path, None)
    }

    /// Load an image and size it accordingly. Also see [`Self::load()`].
    ///
    /// Depending on the filetype and it's parser, the size may be completely ignored.
    /// For other parsers, for example SVG, this attribute determines at what size
    /// the image will be rasterized.
    pub fn load_sized(&mut self, path: &str, size: &Size) -> egui::epaint::TextureId {
        self.internal_load(path, Some(size))
    }

    /// "Completely" unloads the bytes of the image. Also see [`Self::unload_sized()`].
    /// 
    /// If a sized version was allocated, it might sill remain in that cache,
    /// especially if multiple sizes were requested.
    /// If unloading is important, make sure to explicitly unload all (sized) versions
    /// with [`Self::unload_sized()`].
    pub fn unload(&mut self, path: &str) {
        self.bytes_cache.remove(path);
        // TODO: figure out if this can be done without cloning.
        self.texture_id_cache.remove(&(path.to_owned(), (0, 0)));
    }

    /// Unloads **only** a particular size of an image. Also see [`Self::unload()`].
    /// 
    /// The underlying bytes of the image will remain cached.
    pub fn unload_sized(&mut self, path: &str, size: &Size) {
        // TODO: figure out if this can be done without cloning.
        self.texture_id_cache.remove(&(path.to_owned(), *size));
    }

    fn internal_load(&mut self, path: &str, size: Option<&Size>) -> TextureId {
        // Textures that don't have an associated size will be cached with the
        // values (0, 0).
        let size_or_zero = *(size.clone()).get_or_insert(&(0, 0));

        // TODO: figure out if this can be done without cloning.
        if let Some(cached_texture_id) =
            self.texture_id_cache.get(&(path.to_owned(), *size_or_zero))
        {
            return cached_texture_id.clone();
        }

        // TODO: Error handling
        let file_ext = std::path::Path::new(path)
            .extension()
            .unwrap()
            .to_str()
            .unwrap();

        let bytes = self.get_bytes(path);

        // --------------------------------------------------------------------
        //
        // I think it would be better to just cache this, and render it instead
        // of dealing with the weird TextureId / TextureHandle.
        //
        // --------------------------------------------------------------------
        let image = self.parse_bytes(bytes, file_ext, size);

        let texture_id = self
            .egui_texture_manager
            .write()
            .alloc(path.to_owned(), image.into());
        let texture_id = texture_id;

        self.texture_id_cache
            .insert((path.to_owned(), *size_or_zero), texture_id.clone());

        texture_id
    }

    fn get_bytes(&mut self, path: &str) -> Arc<[u8]> {
        if let Some(cached_bytes) = self.bytes_cache.get(path) {
            return cached_bytes.clone();
        }

        let bytes: Arc<[u8]> = std::fs::read(path).ok().map(|bytes| bytes.into()).unwrap();

        self.bytes_cache.insert(path.to_owned(), bytes.clone());

        bytes
    }

    fn parse_bytes(&mut self, bytes: Arc<[u8]>, ext: &str, size: Option<&Size>) -> ColorImage {
        let parser = self
            .bytes_parsers
            .get(ext)
            .expect(&format!("no parser registered for extension {}", ext));

        parser(bytes, size)
    }
}

Performance

I don't really know if this is good performance or not, but here the output of the following code:

let now = Instant::now();

let texture_id = texture_manager.load_sized(PIECE_PATH, &PIECE_SIZE);

println!("allocated texture in: {:.2?}", now.elapsed());

[emilk]

I haven't read through it all (yet), but there are some design constraints that inform the current system:

A) egui::Shape should be serializable, hence the texture ID system. This is important for things like eterm.
B) The egui crate needs to be maximally portable, including to WASM. It can therefor not directly rely on the file system.

Furthermore: the texture system is also a product of evolution, and it can likely be improved further. egui_extras is a proving ground for new things, and we can move things into egui if they mature (and fit within the design constraints).

Registering image loaders is an interesting idea, as it bypasses B). It would also be nice if it could be extended to support e.g. urls.

[Elias-Graf]

Ah, yes. I haven't considered that the shape should be serializable (Is the texture manager also serializable? E.g. will all loaded textures be saved as state?). I still think at some point the rendering should be separated from the texture manager-system.

About loading textures over HTTP Sure! :) And about adding the manager to egui_extras, here you go: https://github.com/Elias-Graf/egui/tree/dynamic-texture-manager.

The manager may be utilized in the following way:

fn ui(self, ui: &mut eframe::egui::Ui) -> eframe::egui::Response {
    const PIECE_PATH: &str = "src/assets/bishop_black.svg";
    const PIECE_SIZE: egui_extras::dynamic_texture_manager::TextureSize = (255, 255);
    const PIECE_SIZE_VEC: egui::Vec2 = egui::vec2(PIECE_SIZE.0 as f32, PIECE_SIZE.1 as f32);

    #[cfg(not(target_arch = "wasm32"))]
    let bytes_loader =
        Box::new(egui_extras::dynamic_texture_manager::bytes_loader::FsBytesLoader::new());

    #[cfg(target_arch = "wasm32")]
    let bytes_loader =
        Box::new(egui_extras::dynamic_texture_manager::bytes_loader::HttpBytesLoader::new());

    let texture_manager = self.texture_manager.get_or_insert_with(|| {
        egui_extras::DynamicTextureManager::new(ui.ctx().tex_manager(), bytes_loader)
    });

    let texture_id = texture_manager.load_sized(PIECE_PATH, &PIECE_SIZE);

    ui.image(texture_id, PIECE_SIZE_VEC)
}

The construction of it is quite messy, I'm hoping for an easier way to acquire it. Possible just configure some default options, and then make it easily adjustable for independent usage.

Is there some error log I could call into? For example when a texture couldn't be found?

Ah, and textures are still never cleared. Is there some sort of way to hook into the painting cycle?

[emilk]

Is there some error log I could call into? For example when a texture couldn't be found?

use tracing

Ah, and textures are still never cleared. Is there some sort of way to hook into the painting cycle?

Cache invalidation is a big part of a texture manager. I don't think you need to hook up to the painting cycle. If the users calls .load* one frame it means the image is in use and should not bee freed.

There are a few strategies to consider

Never free textures:
Only works for apps that only show a few images, and not dynamically load them.

Free textures as soon as they are not used:
This will re-load images that are out-of-sight for a while, which can lead to spike frames when you re-load an image again.

LRU:
Specify a size budget for all images (in bytes) and when we go over the budget, free the least-recently-used textures.