KTX2 + Basis Universal

[pezcode]

I had time to kill so I spent an afternoon investigating how to hook up KtxImporter with Zstandard supercompression and the latest Basis Universal formats. Mainly writing this down so I don't forget again, but also happy to get some input.

Zstandard

This was rather trivial to add to KtxImporter, but might be unneeded (more below).

• zstd has a script to generate single file libraries. The decoder is a ~600kB .c file without dependencies that can be bundled. The readme mentions a compiled WASM binary size of 26kB for the decoder, but I haven't verified that.
• decoding is dead simple, not more than 3 function calls, and it's handled transparently

Updating basisu

Updating basis_universal to 1.15 for BasisImporter worked fairly quickly, too, with two caveats:

• converter tests don't pass because image delta is over the limit, I assume this is the result of some new quality setting somewhere
• keeping compatability with old versions (mainly the prehistoric 1.11 on vcpkg) is a little trickier because some functions and options changed names, but can be worked around with some SFINAE dances (there might be a version define to #if on, haven't checked)

Forwarding Basis-compressed KTX2 to BasisImporter

This one gave me a bit of a headache. I naively assumed BasisImporter could handle the raw data inside KTX2 level data, but currently it only decodes .basis containers. There are ways to use the lowlevel transcoders, but for BasisLZ (Basis ETC1 + LZ supercompression) this is completely insane because you need to validate and decompress the global LZ data and somehow pass that through to BasisImporter.

The far easier solution here is to detect Basis data in KTXImporter early and then directly send the entire file data to BasisImporter. Basisu 1.15 can decode KTX2 and handles all the BasisLZ supercompression stuff for us. BasisImporter then only needs a few minor changes to use ktx2_transcoder next to the existing basisu_transcoder (which only supports .basis containers).

Another effect would be that basisu handles all Zstandard decoding, so that might not have to be handled in KtxImporter, unless it's commonly used outside of Basis-compressed KTX files. Needs some investigation.

[mosra]

Thanks a lot for this!

Updating basisu

Last time I tried updating I got some rather nasty crashes (or ASan failures? don't remember), which is why I put this on the backlog and never looked at it again. If you don't get a crash with 1.15, that's great, the problems solved themselves it seems. Updating the thresholds is completely fine, the images I tested with are extremely small and rather busy so large differences are expected.

Some other tasks that come to mind -- not essential to enabling proper Basis support for glTF, but good to eventually have done as well:

• New Basis has UASTC support in addition to ETC1S. Would be great if the importer/converter plugins were updated to enable that as well. I didn't have time to investigate myself, but I hope it's just about adding some extra formats and (de)compression options, not a whole new API altogether. OTOH I had some nasty surprises with this codebases before, so I don't hold my hopes up.
• IIRC, the encoder/transcoder switched away from std::vector to a non-owning view(?) which could result in some copies being no longer necessary.
• The original encoder code also made heavy use of raw printf() calls that I couldn't override/redirect in any way -- not sure if that changed, but if it did it would be good to have that output redirected to Debug and suppressed if Verbose is not enabled. Batch-processing a lot of images via magnum-imageconverter is extremely noisy right now and this could help a lot.

Forwarding Basis-compressed KTX2 to BasisImporter

I naively assumed BasisImporter could handle the raw data inside KTX2 level data

Yep, my original thought was about doing this manually as well. Since the KTX import is completely under our control and the spec is clear, passing the data to an external tool should be doable -- if the external tool provided a usable interface to consume such data. Which it doesn't, and given the state how feature requests and code contributions are dealt with in the Basis project, I think we shouldn't waste time with this at all. Sigh.

The far easier solution here is to detect Basis data in KtxImporter early and then directly send the entire file data to BasisImporter

That's what I think should be done as well, yes. And as you say, it deals nicely with the compression lib dependency as well. What I'm wondering about is the implications on binary size on the web:

• as far as i can tell, the codebase has its own KTX reader (I was worried at first that it would be depending on Khronos' KTX-Software or some other crazy huge thing), so it could be sufficiently minimal
• the zstd (LZ?) dependency could be turned off, it seems? would be nice to be aware of that in the plugin

Additionally, I need Basis KTX support to handle 2D array images, but since the original .basis files supported that, I hope their KTX reader/writer has that too.

Zstandard

Apart from what Basis handles on its own, I don't have an immediate need for zstd-compressed KTX files and there isn't many KTX files in the wild, so I wouldn't deal with this for now. Yes, the proposed KHR_texture_ktx glTF extension that I want to make use of requires zstd, but since it's still at a proposal stage, there aren't any real glTF+KTX files either, and so I think I could take the liberty of allowing uncompressed files as well.

[Sorry, this is a rather long reply already, but here goes a dump of my research --]

From a brief look at the KTX spec, they don't seem to be doing anything special with the data before sending them to the compressor, which leads me to think that there would be very little size difference between a KTX file with internal zstd compression and a KTX file compressed with a generic zstd library externally. In contrast to various filtering OpenEXR is doing before sending channel data to the compressor it feels like a missed opportunity to me -- and the processing used in OpenEXR isn't anything that would be too complex to be included in the KTX spec.

Given the above, and without dealing with arbitrary external KTX files, I'm thinking KTX files without an internal compression are actually a better storage option:

• Most web traffic is compressed. Assuming browsers

  • would eventually support zstd (e.g. HTTP content-encoding: zstd Fyrd/caniuse#4065) as they did with brotli earlier
  • and ideally having a way to use pre-compressed data on the server so it doesn't have to compress them again for every request,

  the KTX file transparently arrives at the client and from there it can be directly uploaded to the GPU. Without having to inflate the binary size by the decompression library and without spend extra time decompressing the data in a wasm module, which will most definitely be slower than what the browser itself can do (no threading, no advanced SIMD...).

• glTF with embedded uncompressed KTX that gets mmapped is a nice way to "load a huge scene as fast as possible". Or a subset of, certain meshes and certain (slices of) textures. Assuming the whole file

  • could be compressed as a whole with some compression algo that supports seeking (zstd unfortunately not yet, Is zstd splitabble in hadoop/spark/etc?  facebook/zstd#395)
  • and a mmap pagefault handler that could transparently seek and decompress parts of the file that get paged into physical memory (ideas from Filesystem abstraction library corrade#39)

  then this could again be a preferrable option to the usual workflow of having to decompress the whole everything first and then taking out just pieces, leading to huge memory usage spikes. There's also the tradeoff between how fast a SSD can get and how fast the decompression is. If a SSD is large and fast enough it might still be faster to store and load everything uncompressed than having to wait on a slow compression lib, fortunately I think zstd decompression perf still outperforms even the fastest SSDs.

[-- end of the dump.]

Here I don't see any actionable items so far -- unless priorities change, first place where I could see zstd being used is for the filesystem abstraction lib in mosra/corrade#39, and there I would prefer an external dependency, no bundling. 600 kB is quite a lot -- that's the size of json.hpp we now happily made obsolete with #107 :)

[pezcode]

I got some rather nasty crashes (or ASan failures? don't remember)

Good point, I'll check again with ASan

UASTC support [...] I hope it's just about adding some extra formats and (de)compression options, not a whole new API altogether

From a quick glance that's done transparently by the transcoder, and the encoder only needs to flip a single bool to get UASTC output.

raw printf() calls that I couldn't override/redirect in any way

That's still there, with no way to override it. It has a macro BASISU_DEVEL_ERROR but it always defines that, so no way to redirect it even in source builds.

I need Basis KTX support to handle 2D array images

2D array images are supported: https://github.com/BinomialLLC/basis_universal/blob/master/transcoder/basisu_transcoder.cpp#L16803

[mosra]

Oh, forgot to reply on this part:

keeping compatability with old versions (mainly the prehistoric 1.11 on vcpkg) is a little trickier

In most packages (Arch, Homebrew, and even vcpkg) I'm directly fetching a particular commit from the Basis repo and building against that. Because Basis is such a janky project and it has no real buildsystem nor anybody cared to create any packages for any system, I think it's fine if you just upgrade to whatever tag/version/? is latest without keeping backwards compatibility -- the majority of people who's using the packages won't notice anything as the package transparently fetches the new Basis version, and people using Magnum as a subproject will just update the Basis submodule if they discover a breakage.

I went through this compatibility pain for SPIR-V Tools and glslang because those are essential projects and package managers have a ton of different versions against all of which it should compile. And there it's a hell because neither really has a version define, so it was a ton of try_compile() misery.

But Basis is a rather exotic thing (especially given that there are no packages for it at all whatsoever), so unless the compatibility is just a matter of a few trivial #ifs against some version define in some header (which, given the project history, I seriously doubt), it's a waste of time.

[mosra]

Done with #112 and #113 🎉