GLTFLoader: Implement Materials Variants extension handler as plugin

[takahirox]

Related PR: #20690

Description

glTF Materials variants extension example has been added in #20690. It's a nice example indeed.

Currently KHR_materials_variants extension handler is implemented in examples/webgl_loader_gltf_variants.html. Handling the extension without the loader change would be good energy savings. And as Don mentioned, no variants abstraction in Three.js. So handling in tools or apps side would sound reasonable.

On the other hands, there are some concerns. Users who want to use the extension need to copy and paste the extension hander from the example to their applications. Or they need to be aware of the extension specification, learn the parser API, and write their own similar handlers.

Suggestion

What do you think of providing basic variant select function from the loader and implementing the handler as plugin? And also we keep holding the extension information in .userData for users want to write their variant select function? (And they can be used when exporting.)

I tried it and open a PR as draft. I want you folks to review and to give the feedbacks especially about

1. Is providing such function from the loader good?
2. Are the suggested APIs good?

API

// gltf.userData.variants: Array<strings>
// gltf.userData.selectVariant(variantName: string): Promise<>

import GLTFMaterialsVariantsExtension from './jsm/loaders/gltf_plugins/KHR_materials_variants.js';

const loader = new GLTFLoader();
loader.register(parser => {
  return new GLTFMaterialsVariantsExtension(parser);
});
loader.load('model.gltf', async gltf => {
  scene.add(gltf.scene);

  const variants = gltf.userData.variants; // ['variantName0', 'variantName1', ...]
  await gltf.userData.selectVariant(variants[0]); // switch to variant material of the meshes under gltf.scenes
  render();
});

Another option would be adding variant select helper static function as GLTFLoader.Utils.selectVariant or somewhere else.

Plugin API change

1. Add invokeAll beforeRoot() and afterRoot(result) hook points. They are especially good for handling extensions under the glTF root. I assume very basic use cases are beforeRoot for hacking gltf definitions and afterRoot for hackinig generated objects. KHR_materials_variants extension handler uses only afterRoot but I think beforeRoot would be useful, too. I actually needed it when I tried VRM loader. Regarding the performance overhead, one hook point each before and after entire glTF file parse so I haven't measured the performance yet but I think it won't be big response time overhead.

2. Add addExtensionsToUserData parameter to plugin. (Better name idea is welcome.) If it's true, extension information are saved under foo.userData as we do for unknown extensions.

I think these additions would be worth even if we will end up concluding we won't implement the extension handler as plugin.

/cc @donmccurdy

[takahirox]

Note: I save variants in scene.userData.variants, too, because they can be used for future glTF exporter support. Otherwise, variants parameter may need to be separately passed to the exporter.

[takahirox]

Taking scene as a parameter instead of using gltf.scenes would be another option.

[donmccurdy]

Sounds OK to implement the variant example using the plugin API, if it's not much more complicated for users to understand when reading the example. But I'd prefer that the plugin itself remain in the example, and not ship as part of GLTFLoader... I think it's important that GLTFLoader returns something that behaves like a normal THREE.Group hierarchy, and having GLTFLoader return functions like selectVariant crosses that boundary a bit more than I'd like. Variants are an abstraction that does not really fit in three.js — it's important that users are able to implement variant-like behaviors using three.js, but not that three.js provide a specific variant system out of the box.

The before/after root hooks sound like good simple additions. Is the addExtensionsToUserData one necessary, or could that be implemented another way?

[takahirox]

Thanks for sharing your opinion. It makes sense to me. GLTFLoader is almost core code (for me). Having the loader support only the features fitting to native Three.js, and putting the others in example but not in the loader, sounds good.

I made examples/jsm/loaders/gltf_plugins directory and placed the KHR_materials_variants handler in it. (Let me know any better places if you have.) Users can reuse the handler in their applications like.

import { GLTFLoader } from './jsm/loaders/GLTFLoader.js';
import GLTFMaterialsVariantsExtension from './jsm/loaders/gltf_plugins/KHR_materials_variants.js';

const loader = new GLTFLoader().register(parser => {
  return new GLTFMaterialsVariantsExtension(parser);
});
loader.load( 'MaterialsVariantsShoe.gltf', async gltf => {
  scene.add(gltf.scene);
  const variants = gltf.userData.variants;
  await gltf.userData.selectVariant(variants[1]);
  render();
});

If users want their own variant select function, they don't register the handler and they can write their own function with the data under the .userData.gltfExtensions['KHR_materials_variants'] like the current examples/webgl_loader_gltf_variants.html does.

Is the addExtensionsToUserData one necessary, or could that be implemented another way?

selectVariant() function needs the sets of glTF mesh definition and generated Three.js Meshes. But after the loader parses the glTF, there is no way to find a gltf mesh definition corresponding to a Three.js mesh generated by the loader. So, saving the extension parameters in mesh.userData when the meshes are created would be the easiest solution.

The loader saves the parameters for unknown extensions. If the KHR_materials_variants plugin is registered, it'll be a known extension and the loader doesn't save the parameters. New plugin parameter addExtensionsToUserData = true lets the loader saves the parameters even though the extension is known by the loader.

Another solution for saving the parameters of known extension may be adding invokeAll afterXXX hook points.

[elalish]

I'm interested in this for model-viewer as well (I already added variants support based on your example). However, I need the ability to export those variants as well with GLTFExporter, which currently fails because though it retains the variants userData, it is missing any materials that aren't currently in the scene graph and the index values no longer match what is output. I was thinking the solution to this might involve the associations member of the parser object, which will hold references to those extra materials once they get parsed. I'm wondering if the associations might also help with this PR?

I think I could implement variants export now using onCompleted for a glTF, but if it exports a GLB, it seems onCompleted is too late to actually affect the JSON output. I'm wondering if this is a bug or if we need a different way to hook into the exporter. Thoughts @donmccurdy?

[elalish]

Ah, just saw #20842, which sounds promising!

[mrdoob]

gltf.userData.selectVariant(variants[1]);

Adding methods to userData feels hacky to me...

What do you guys think about creating a GLTFMesh class in GLTFLoader that extends Mesh?
GLTFMesh could have GLTF specific properties and methods which GLTFExporter would be able to parse nicely.

[takahirox]

@elalish Yeah, my original motivation of #20842 is I want to export materials variants.

@mrdoob GLTFMesh sounds good to me, I haven't deeply thought yet tho. GLTFLoader creates Mesh or SkinnedMesh then we may need both GLTFMesh and GLTFSkinnedMesh.

[donmccurdy]

What do you guys think about creating a GLTFMesh class in GLTFLoader that extends Mesh?

I am pretty nervous about this... I would prefer that GLTFLoader produced normal three.js objects. From my perspective it's OK if we don't support KHR_materials_variants — i.e. don't provide any official variant-switching API — as long as users can still support it on top of three.js relatively easily.

it is missing any materials that aren't currently in the scene graph ... I was thinking the solution to this might involve the associations member of the parser object, which will hold references to those extra materials once they get parsed.

Note that GLTFLoader does not process materials until they're needed for something, either by an object in the scene graph, or direct request by an external call to getDependency.

I think I could implement variants export now using onCompleted for a glTF, but if it exports a GLB, it seems onCompleted is too late to actually affect the JSON output. I'm wondering if this is a bug or if we need a different way to hook into the exporter. Thoughts @donmccurdy?

I don't think I quite understand how onCompleted would be used here, but if we need an additional hook to support this extension that's fine with me.

[elalish]

@donmccurdy Fair, after reviewing the code a bit more I was mistaken about what onCompleted was for. What I'm really looking for is #20842, which seems like it would have what I need. If we want to keep variants support in plugins, then I suppose the Loader plugin could just add a Map<Variant, Material> member to the Mesh.userData and the Exporter plugin could then read them. However to be useful we still need some top-level functions for selecting a variant, perhaps adding and removing a variant. Where should these live? Should the plugin add them to the gltf object?

[takahirox]

What do you guys think about creating a GLTFMesh class in GLTFLoader that extends Mesh?

I am pretty nervous about this... I would prefer that GLTFLoader produced normal three.js objects.

Yeah, it reminds me of that I agree with GLTFLoader generate normal Three.js objects and then I suggested implementing handlers of the extensions requiring beyond the Three.js core APIs out of the loader. So GLTFMesh would be opposed to this idea.

However to be useful we still need some top-level functions for selecting a variant, perhaps adding and removing a variant. Where should these live? Should the plugin add them to the gltf object?

I'm thinking of placing an unofficial function selectVariant() to glTF scenes. And some core APIs like scene.copy/clone() should know the method so the materials variants plugins
may need to override them of the glTF scenes.

[mrdoob]

I'm thinking of placing an unofficial function selectVariant() to glTF scenes. And some core APIs like scene.copy/clone() should know the method so the materials variants plugins
may need to override them of the glTF scenes.

Adding variants code in scene.copy/clone() doesn't sound good to me (at least not yet). If we did a GLTFMesh (extends Mesh) then we would only need to add variants code in GLTFMesh's clone().

[elalish]

Technically the variants aren't defined at the glTF scene level, but at the glTF root level. Therefore I would propose putting selectVariant() on the GLTF object itself, which doesn't require any changes to copy/clone(). No methods are needed on Mesh and it already copies userData, but unfortunately it stringifies it so it won't retain object references. So yeah, looks like we'll need GLTFMesh with a copy() method that brings along the new Map member.

[mrdoob]

Ah, sorry...
Considering it's the glTF root level, I guess it should not be GLTFMesh but GLTFObject3D (extends Object3D) maybe?

[takahirox]

How about adding selectVariant() function to GLTFMaterialsVariantsExtension plugin?

import GLTFMaterialsVariantsExtension from './jsm/loaders/gltf_plugins/KHR_materials_variants.js';

const loader = new GLTFLoader();
loader.register(parser => {
  return new GLTFMaterialsVariantsExtension(parser);
});
loader.load('model.gltf', async gltf => {
  scene.add(gltf.scene);

  const variants = gltf.userData.variants; // ['variantName0', 'variantName1', ...]
  await GLTFMaterialsVariantsExtension.selectVariant(gltf.scene, variants[0], true /* doTraverse */);
  render();
});

In this PR GLTFMaterialsVariantsExtension plugin saves the mappings of variant-material under .userData. GLTFMaterialsVariantsExtension.selectVariant(object, variant, doTraverse) traverses a scene and selects a material associated with variant from .userData for each object.

The .userData are automatically copied in clone/copy() so no need to hack clone/copy(). And no need of GLTFMesh/Object3D. And the exporter materials variant plugin with #20842 will check the data under .userData and can export the materials variant extension.

A problem is serialization from/to json wouldn't work for material variants extension because Three.js json serialization doesn't serialize Three.js objects/references in .userData. GLTFMaterialsVariantsExtension may provide helper functions for json serialization if needed.

[donmccurdy]

loader / gltf / variants is really a pretty short and simple example, does it need to be different than it is? There are other ways the application might want to handle its variants, like activating a variant only for specific objects in the scene. I think I would rather leave all of that in application-land.

If we need to change something to support exporting variants, we can do that on its own, too.

[takahirox]

loader / gltf / variants is really a pretty short and simple example, does it need to be different than it is?

I personally think yes as I wrote in the PR comment. The code in loader / gltf / variants is not so long. But users manually need to copy and paste the code to their applications. And to understand the code may be a bit hard especially for new or non-JS devs like artists because it requires to know the parser API, the fact that data of the extensions which the loader doesn't know are placed under .userData, and material variant extension spec.

GLTFMaterialsVariantsExtension plugin hides such difficulty from users. Users will need to just import the plugin from a CDN server, register to their loader, and call selectVariant() function. I think it would be much easier to them.

On the other hand, I also can understand Don's feeling that loader / gltf / variants example may be good enough as Three.js official examples. If we don't want to have the plugin in Three.js repository I will put it in my repository. And I would like to make another PR to add before/afterRoot hookpoints and addExtensionsToUserData plugin parameter the plugin needs instead.

[takahirox]

I made a PR #21207 for before/afterRoot hook points. If it'll be merged, I can make the plugin and start to place it on my repository.

Regarding addExtensionsToUserData parameter, I realized that the plugin can save the extension data (in afterRoot) even without introducing addExtensionsToUserData parameter because the plugin can find a associated gltf.primitive with each generated Three.js Mesh by using parser.associaions.

[donmccurdy]

#21207 looks good to me, thanks.

I do still prefer that GLTFLoader should return plain three.js objects, and not glTF-specific methods or subclasses. Both are hard to serialize or clone, and increase the public API surface and maintenance responsibilities related to GLTFLoader.
Extensions that are not attached to GLTFLoader by default, or in another repo, would be OK even if we need added hooks like #21207.

[takahirox]

I wrote the KHR_materials_variants plugins for GLTFLoader and GLTFExporter and released in my repository.

https://github.com/takahirox/three-gltf-plugins

Let's consider moving them to the Three.js repository if I (or we) come up with an idea to make the extension great fit to plain Three.js APIs and structures.

BTW I guess @elalish may be interested in the exporter plugin?

[donmccurdy]

Nice work @takahirox! Do you want to put a link next to the extensions listed in https://threejs.org/docs/#examples/en/loaders/GLTFLoader?

[takahirox]

That sounds good. I would do soon, after I have post-surgery health checkup today.

[takahirox]

@donmccurdy

Opened a PR for the doc #21261 I would be happy if you review. And I also pleased if you comment for #19359 (comment) because the suggestion can simplify my materials variants plugin.