Compress textures when importing a GLTF2 that has built-in textures as BasisU

[fire]

Describe the project you are working on

Online 3d game.

Describe the problem or limitation you are having in your project

We wrote a VRM importer based on the GLTF importer.

Given a single VRM (derivative of GLB) https://github.com/Miraikomachi/MiraikomachiVRM/blob/master/Miraikomachi.vrm.

Previous tests in Godot 3 saw the renderer decompress pngs every frame and cause problems.

Even if the decompression doesn't happen, storing the images in an ImageTexture is still large and not optimized for the gpu.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Convert ImageTextures built-ins to basis universal built-in textures for size and speed gains.

Why BasisU? BasisU allows the same format to be used in Web (etc2) and PC (bptc, dxt5).

// Open problem is that it currently uses the developer's file formats rather than the gamer's capabilities.
Ref<ImageTexture> t;
t.instantiate();
if (true) {
	img->resize_to_po2();
	img->generate_mipmaps();
	Image::CompressSource csource = Image::COMPRESS_SOURCE_GENERIC;
	Image::UsedChannels used_channels = img->detect_used_channels(csource);
	Ref<StreamPeerBuffer> buffer;
	buffer.instantiate();
	// Note: Update when the CompressedTexture2D format changes.
	buffer->put_32(CompressedTexture2D::DATA_FORMAT_BASIS_UNIVERSAL);
	buffer->put_16(img->get_width());
	buffer->put_16(img->get_height());
	buffer->put_32(img->get_mipmap_count());
	buffer->put_32(img->get_format());
	for (int i = 0; i < img->get_mipmap_count() + 1; i++) {
		Vector<uint8_t> data = Image::basis_universal_packer(img->get_image_from_mipmap(i), used_channels);
		int data_len = data.size();
		buffer->put_32(data_len);
		const uint8_t *r = data.ptr();
		buffer->put_data(r, data_len);
	}
	PackedByteArray array = buffer->get_data_array();
	FileAccessMemory *f = memnew(FileAccessMemory);
	f->open_custom(array.ptr(), array.size());
	Ref<Image> placeholder = CompressedTexture2D::load_image_from_file(f, 0);
	t->create_from_image(placeholder);
	memdelete(f);
} else {
	t->create_from_image(img);
}
state->images.push_back(t);

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Discussion with reduz mentioned we can replace ImageTexture usage with BasisU.

Create a resource for BasisU. We can add options per importer now.

If this enhancement will not be used often, can it be worked around with a few lines of script?

This can be worked around with a page of code, but it is also a bug in Godot Engine.

Is there a reason why this should be core and not an add-on in the asset library?

This is a also a bug in core Godot Engine.

[fire]

I encountered some problems exploring this proposal. Will report back.

[fire]

Ref<ImageTexture> t;
t.instantiate();
img->generate_mipmaps();
Image::CompressMode mode = Image::COMPRESS_MAX;
// Manually guess the image compression mode.
if (ProjectSettings::get_singleton()->get("rendering/textures/vram_compression/import_bptc")) {
	mode = Image::COMPRESS_BPTC;
} else if (ProjectSettings::get_singleton()->get("rendering/textures/vram_compression/import_s3tc")) {
	mode = Image::COMPRESS_S3TC;
} else if (ProjectSettings::get_singleton()->get("rendering/textures/vram_compression/import_etc2")) {
	mode = Image::COMPRESS_ETC2;
} else if (ProjectSettings::get_singleton()->get("rendering/textures/vram_compression/import_etc")) {
	mode = Image::COMPRESS_ETC;
}
Image::CompressSource csource = Image::COMPRESS_SOURCE_GENERIC;
Image::UsedChannels used_channels = img->detect_used_channels(csource);
img->compress_from_channels(mode, used_channels, 0.7f);
t->create_from_image(img);
state->images.push_back(t);

This code is simpler.

How to use variants?

[Calinou]

Basis Universal is very slow to compress (and quality isn't the best), so this should be an opt-in import option to avoid impacting iteration times negatively. For most use cases, standard VRAM compression remains a better fit in my opinion.

[fire]

Yes. The problem now remains how to load the variations of the CTEX from a ImageTexture. @Calinou any ideas?

Like S3TC and ETC2 or ETC2 and (the unimplemented ASTC).

I'm going to abandon this proposal in a bit.

[lyuma]

Respectfully, I disagree with the proposal.

In short, this is trading one problem for another: While it does fix imported glb textures which are currently broken (imported as raw), it means that imports from glb will be forced into the BasisU path.

In my opinion, images embedded in a glb or fbx should behave no different from images imported separately.

If we do move forward with this proposal, I would like to pose the question: why are all texture imports not handled only as BasisU? My assumption is there is a reason. For example the arguments brought up by @Calinou : compression time and quality.

I would rather focus on options such as extracting embedded textures into the project or a specialized .godot folder, or another way to trigger the standard image compression paths.

[reduz]

@lyuma I think these are two unrelated issues.

The main problem here is when you have a GLTF with built-in textures. If this happens during import, you may be able to choose whether you want to just keep these files built-in, or extract the textures automatically (can be an option in the GLTF importer). If you don't want to extract the textures then the texture importer will not run. The result will not be portable when exported (Remember that Godot almost does not run logic on export, it always does everything on import).

The alternative to BasisU is having two different texture formats, one for mobile and one for desktop but this takes a large amount of space. One alternative for this PR could be to also specify which kind of compression you want for textures besides basis universal (BC, BPTC, ETC, ASTC, etc) but of course remember that this is not portable (which may not be a problem if you are not making a multi platform game).

[reduz]

@fire If you are OK we can do the following path:

• I change the PR to also include other compression types (BC, BPTC, S3TC - we dont support ASTC yet, can be added later -).
• You add in the GLTF import options what do you want to do with built-in textures (choose extract files, or one of the PortableCompressedTextures formats)

Extract files is not simple though, but I can help you figure it out.

[lyuma]

In short, my biggest concern was that BasisU wouldn't be fast or good enough quality, meaning godot would need an option for conventional vram compression.

To that end. Fire has been very convincing that the quality of BasisU has improved greatly in the new version... so we should proceed under the assumption that BasisU is good enough.

I would love to have the option to extract textures, which would give an alternative if the portable texture format isn't suitable or the user wants more fine-grained control per-texture.

We should continue testing BasisU on various scenes to see if or how often compression artifacts occur. We should also keep our eyes on the speed of compression, since this will be added to the import time of any .glb model.

[fire]

I'll ask @lyuma for details, but I think the extract files codepath may take more time to engineer and since it's an enum addition, we shouldn't block on adding the option to PortableCompressedTextures since we can add it later.

We might not even need the extract files feature.

[fire]

Also @reduz I searched the BasisU code base and it still doesn't support HDR images.

[lyuma]

I agree on the "Basis" of fire's testing and PRs to upgrade and improve the basis UASTC settings. Basis compression seems quite capable as a portable texture format.

Regarding my original objection that this puts embedded textures on a different codepath from separated textures, I had a nice discussion with reduz, and I understand that permitting textures to be extracted to an intermediate directory and reimported would add needless complexity to the import pipeline for what seems to be very little gain.

For these reasons, I am in favor of the proposal and the design of PortableCompressedTexture.

Finally about texture extraction, I still think that would be useful to support but it should be done separately and needs its own proposal.

[lyuma]

@fire about HDR and RGBAF, I can't find which glTF extension defines .exr / HDR support. what is the usecase for needing HDR texture support here?

Perhaps If needed, HDR textures can be automatically extracted if there is no proposal for a Portable (basis) HDR texture, given that embedded exr in gltf are not possible without an extension

[fire]

@lyuma KhronosGroup/glTF#1955

The typical usecase for HDR is image based lighting.

[fire]

Reopening because, there's further work to be merged.

[fire]

Todo:

Add GLTF import built-in textures options (Choose one of the PortableCompressedTextures formats) instead hardcoding basis.