Supporting archive assets

[viridia]

What problem does this solve or what need does it fill?

From the discord: @cart "Ok I'm reasonably convinced that we need a better answer for "archive style" assets. Maybe we need a per-asset configuration that defines how root asset lifetimes work".

Terms: An "archive" asset in this context is a type of bundle file whose purpose is to contain other assets. Zip files are one example, but this pattern is also often seen in JSON, MessagePack and GLTF files.

There are two kinds of archives: seekable, and non-seekable. Seekable archives are ones where it is possible to load an individual record without reading the entire file serially, by seeking directly to the start of the record. Zip files are an example of this: they contain a directory index which contains the file offset of every entry. I think that this use case is already adequately supported in Bevy, using a custom AssetReader, so I don't think we need to do anything.

Non-seekable archives are ones where you have to read the whole thing in order to load any part. JSON, MessagePack, and CBOR are examples. These are currently problematic in Bevy.

Often a developer will want to bundle together related assets to minimize the number of i/o operations: if the assets are all loaded together, then the overhead of opening and reading a file need only be performed once instead of opening a file for each asset individually.

Unfortunately, in current Bevy doing this can actually result in worse performance instead of better. The reason is due to the way asset lifetimes work. The asset system produces a handle for both the container and for each labeled asset, but if you only hold on to the handle for the labeled asset, then the container is immediately dropped. If you then want to subsequently load another labeled asset from the same archive, the asset system has to re-load and re-parse the entire container all over again.

Note that the word "subsequently" in the preceding paragraph has two sub-cases: one is loading two labeled assets from the same container at the same time. The other is loading a labeled asset, and then loading a second labeled asset from the same container at a later time, but while the first handle is still being held.

I'm not sure what exactly is Bevy's behavior in either of these cases: the docs don't say. But in either case, it would be more efficient if we could keep the container asset in memory long enough to retrieve the second asset.

A third possible case is caching, in which a handle is loaded, dropped, and a second handle from the same container is loaded shortly thereafter. However, I think that this case is probably out of scope. Developers can accommodate this case by maintaining an LRU of recently-loaded handles.

Finally, there is the need for keeping the container in memory for purposes of editing. Non-seekable assets cannot be written to disk without writing the whole thing: in order to modify one record, you need to load the entire archive, modify individual records, and then write out the whole thing.

For all of these use cases, the solution is to provide a way to keep the entire container in memory as long as any individual labeled asset is held.

A naive solution to the problem is to design the data type for labeled assets to store a strong handle to the container. Unfortunately, this creates the possibility of a "reference cycle" such that the container will never be freed.

Also, any solution needs to accommodate the case of records which themselves contain dependencies/handles to other records in the same container, or records in other containers: reference cycles should not be created unless the dependencies are actually cyclic.

What solution would you like?

I'm not going to specify a specific solution here yet, because I want to collect feedback on possible solutions. But the general form of the solution should be a way to keep containers in memory as long as a handle is held. This should be an option in the asset loader implementation.

What alternative(s) have you considered?

I've played around with storing additional handles in the labeled asset, but I worry about creating reference cycles by doing this.

[viridia]

A couple more thoughts on this:

For purposes of editing, it will be useful to be able to iterate through all the entries in the archive. For example, a user may want to select items from a catalog via an interactive chooser ui, which will need a way to iterate the entries.

As far as I know, currently in Bevy labeled assets don't currently maintain any sort of relationship with the root asset after loading, so this would need to be implemented by the archive asset type.

The obvious way to do this is to have the root asset maintain a hash map of all the entries by label, however it's not clear whether this should be a map of handles or a map of arcs: that is, should the catalog's internal directory go through the asset handle mechanism, or should it hold on to the entries in a more direct way? As mentioned previously, we want to avoid creating cyclic handle dependencies where possible.

Also for purposes of editing, there should be some means of gaining access to the catalog given a handle to an entry; however I think this can be done via manipulation of the asset path - strip off the label portion and then get the handle from the asset server via .load(). The reason this is needed is to be able to decide what needs to be saved when an entry is modified.