Support for KHR_materials_variants in cesium-native and implementations

[javagl]

tl;dr Summary:

• The KHR_material_variants extension allows different material configurations for glTF assets that can be selected at runtime
• The same could be applied to a tileset, to allow selecting a material variant for all its glTF tile contents
• The information about the variants would be stored in the tileset JSON (e.g. as metadata)
• Support for processing this material variant information and displaying the selected variant will have to be added to the runtime engines, e.g. cesium-unreal and cesium-unity

---

Goal

KHR_material_variants is a ratified glTF extension that allows defining different material variants for meshes:

• The extension defines a set of "variant names" (identified by their name) at the top level of the glTF asset
• For each mesh primitive, the extension defines the material that should be used for this primitive when a certain variant is selected

It should be possible to apply the concept of "material variants" to the more coarse-grained level of tilesets. The tileset should contain information about the variants that are expected to be present in the glTF assets that it refers to. Runtime engines should offer a mechanism to switch between these variants (i.e. the materials).

cesium-native glTF level

There already is very basic support for KHR_materials_variants in CesiumGltf, as of #630 : This PR added the classes that are auto-generated from the schema, allowing to obtain the top-level information about the available variants, and the mapping between materials and variants for each primitive.

cesium-native tileset level

There could be information about the avaialable material variants for a tileset, that summarizes the variants that are available in the glTF assets that are used as the tile content of the tileset.

In the most simple case, there could be an array of (unique) strings that match the names of the material variants that are supposed to be contained in the glTF assets. The user should then be able to select one of these material variants, and this variant should be activated for all glTF assets. When a glTF does not contain the respective variant name, then it should be rendered with its default material.

Some technical details still have to be sorted out. But the rough idea is to store the information about the variants as Tileset- or Group metadata. So the information could be stored as

{
  "schema": {
    "classes": {
      "materialVariants": {
        "properties": {
          "material_variants": {
            "type": "STRING",
            "array": true,
            "description": "Names of material variants to be expected in the glTF assets"
          }
        }
      }
    }
  }
}

Depending on the level of integration, there could be "convenience functions" for accessing this information. Specifically: It would be preferable if cesium-native offered an infrastructure where it is not necessary to manually search through the schema and check for the presence of certain properties and extract data from property tables, but just say
std::vector<str::string> variantNames = Magic.getThemFrom(tileset);

Where exactly the support for metadata in general (and the variants in particular) should be added still has to be decided.

• Basic infrastructures for reading metadata are in Cesium3DTiles
• In order to be useful for runtime engines, it has to be offered in Cesium3DTilesSelection

Runtime engine level

Once the basics have been sorted out, it has to be decided how runtime engines could process this information. Regardless of how the information is transported from the tileset.json into the runtime engine, the engine will have to offer things like a UI component for selecting the variant, and (and that may be the most challenging part) switch between different materials, for all tile contents that are currently loaded.

Collecting approaches (and possible constraints) for the implementations in each runtime engine can be done here, and moved into issues of the respective engine implementations when the overall approach as been agreed on.

Support in cesium-unreal

I started zooming into the relevant code paths, for the example of cesium-unreal. But this was really only a first pass. High level thoughts:

• There is the IPrepareRendererResources interface
• It receives a CesiumGltf::Model instanance, and generates/fills the engine-specific representation (i.e. a UCesiumGltfComponent)
• The UCesiumGltfComponent already has a BaseMaterial, which is the engine-specific UMaterialInterface instance
• It also has a BaseMaterialWithWater and BaseMaterialWithTranslucency. This already seems to be some sort of a "material variants"...
  -> Maybe the material variants could be represented in a similar way?

The variants could be stored as an array of UMaterialInterface objects. The Cesium3DTileset could have a method like setVariant(std::string variant) that activates the respective material for all the glTF assets. All this should be possible without reloading the whole tileset (otherwise it would defeat the purpose of the extension...), but ... I know that this can be tricky.

[kring]

There are some possibly-relevant ideas about switching materials based on metadata in this Unreal issue:
CesiumGS/cesium-unreal#771

[javagl]

The linked issue sounds like it could go far beyond what is required for KHR_materials_variants. In this extension, all the materials are known at load time of the glTF asset, and they are standard glTF materials (no custom shaders or properties). But I'm certainly not up to date with the state of the implementation, or the capabilities of the broad topic of 'layers' in CfU.

---

Other than that, I started creating a simple test data set. This is only intended as a DRAFT, while the technical questions are still being sorted out here.

It is an asset that contains two glTF assets, each having material variants RGB, RRR, GGG, and BBB. The names ... do not seem to be overly creative, but ... that's the asset:

The tileset defines

1. tileset-level metadata (with all variant names)
2. two groups with metadata (with 2 variant names each)

There is a sandcastle that pragmatically extracts that metadata, and puts it into a UI, so that the variants can be selected:

The exact behavior and some corner cases have to be clarified. For example: Will there only be either tileset-level metadata or group metadata (or could there be both, and how is that supposed to be handled?). Beyond that, there should also be a test data set where the glTF assets have different sets of material variants, just to see how this can be handled sensibly.

material-variants-2023-06-23.zip

[javagl]

A first, early draft PR is at #693

The current idea is to have a simple structure, called MaterialVariants

struct MaterialVariants {
  std::vector<std::string> tilesetMaterialVariantNames;
  std::vector<std::vector<std::string>> groupsMaterialVariantNames;
};

This would then be obtained from the Cesium3DTilesSelection::Tileset, as in

const MaterialVariants& getMaterialVariants() const noexcept;

This structure is filled from the metadata that is contained in the tileset.json. Specifically, it is filled from the information that is given in the tileset.json via

  "metadata": {
    "class": "MaterialVariants",
    "properties": {
      "material_variants": ["RGB", "RRR", "GGG", "BBB"]
    }
  },
  "groups": [
    {
      "class": "MaterialVariants",
      "properties": {
        "material_variants": ["RGB", "RRR"]
      }
    },
    {
      "class": "MaterialVariants",
      "properties": {
        "material_variants": ["GGG", "BBB"]
      }
    }
  ],

NOTE: In the ZIP that is attached to the previous comment, the tileset only defined a single class with a single property for storing these material variants. With that alone, there is no way to determine whether someting is really a "material variant", or just some string array. Therefore, I think that it will be necessary to add a semantic to the property. For now, this is done with...

...
         "material_variants": {
            "type": "STRING",
            "array": true,
            "description": "Names of material variants to be expected in the glTF assets",
 
             // This line:

            "semantic": "MATERIAL_VARIANTS"
          }
...

but the naming and other details still have to be sorted out.

[javagl]

The approach for accessing the material variants information based on #709 is shown in #693 (comment)

[javagl]

This was completed with #709 . (This only covers the cesium-native side of it. For support in the runtime engines, dedicated issues can be opened).

The linked PR adds support for general metadata on the tileset level.

The following is an example of how the material variants information can be accessed. It is wrapped into a small utility class for convenience here:

struct MaterialVariants {
  std::vector<std::string> tilesetMaterialVariantNames;
  std::vector<std::vector<std::string>> groupsMaterialVariantNames;
};

class MaterialVariantsUtilities {

public:
  static CesiumAsync::Future<MaterialVariants>
  fromTileset(Cesium3DTilesSelection::Tileset &tileset) {
    return tileset.loadMetadata().thenImmediately([](auto &&pMetadata)
                                                      -> MaterialVariants {
      MaterialVariants materialVariants;
      materialVariants.tilesetMaterialVariantNames =
          MaterialVariantsUtilities::findStringPropertyValues(
              *pMetadata->schema, *pMetadata->metadata,
              MaterialVariantsUtilities::MATERIAL_VARIANTS_SEMANTIC_NAME);
      for (const Cesium3DTiles::GroupMetadata &group : pMetadata->groups) {

        materialVariants.groupsMaterialVariantNames.push_back(
            MaterialVariantsUtilities::findStringPropertyValues(
                *pMetadata->schema, group,
                MaterialVariantsUtilities::MATERIAL_VARIANTS_SEMANTIC_NAME));
      }
      return materialVariants;
    });
  }

  static void debugPrint(const MaterialVariants &materialVariants) {
    std::cout << "Material variants:" << std::endl;
    std::cout << "  Tileset:" << std::endl;
    const auto &t = materialVariants.tilesetMaterialVariantNames;
    for (size_t i = 0; i < t.size(); i++) {
      std::cout << "    " << t[i] << std::endl;
    }
    std::cout << "  Groups:" << std::endl;
    const auto &gs = materialVariants.groupsMaterialVariantNames;
    for (size_t i = 0; i < gs.size(); i++) {
      std::cout << "  Group " << i << ":" << std::endl;
      const auto &g = gs[i];
      for (size_t j = 0; j < g.size(); j++) {
        std::cout << "    " << g[j] << std::endl;
      }
    }
  }

private:
  static inline constexpr const char *MATERIAL_VARIANTS_SEMANTIC_NAME =
      "TILESET_MATERIALS_VARIANTS_NAMES";

  static std::vector<std::string>
  findStringPropertyValues(const Cesium3DTiles::Schema &schema,
                           const Cesium3DTiles::MetadataEntity &metadataEntity,
                           const std::string &semantic) {
    std::optional<Cesium3DTiles::FoundMetadataProperty> propertiesWithSemantic =
        Cesium3DTiles::MetadataQuery::findFirstPropertyWithSemantic(
            schema, metadataEntity, semantic);

    const CesiumUtility::JsonValue::Array &propertiesJson =
        propertiesWithSemantic->propertyValue.getArray();
    std::vector<std::string> propertyValues(propertiesJson.size());
    std::transform(propertiesJson.begin(), propertiesJson.end(),
                   propertyValues.begin(),
                   [](const CesiumUtility::JsonValue &value) {
                     return value.getStringOrDefault("");
                   });
    return propertyValues;
  }
};

From a given Cesium3DTilesSelection::Tileset, the material variants information may then be fetched and printed as follows:

  auto &&future =
      MaterialVariantsUtilities::fromTileset(tileset).thenImmediately(
          [](MaterialVariants &&materialVariants) -> void {
            MaterialVariantsUtilities::debugPrint(materialVariants);
          });

(Note that this future will be resolved only after the tileset JSON has been fetched and processed. So it will be resolved after one of the updateView... family of functions on the tileset has been called).

Here are two very basic examples for testing - it is the same tileset and the same GLBs as above, once with a schema in the tileset, and once with an external schema:

materialVariantsExamples-2023-09-08.zip