Allow access to Tileset-level metadata (schema, schemaUri, groups, metadata)

[kring]

This is a PR into #711 so merge that first.

Fixes #676

This PR adds the ability access the schema, schemaUri, groups, and metadata properties in a top-level tileset.json as well as in an external tileset's tileset.json. For the top-level tileset.json, this information is stored on the root tile, so it's not available until the root tile exists. It's captured in a new type called TilesetMetadata, and the easiest way to obtain it is from the Tileset:

Tileset tileset = ...;
const TilesetMetadata* pMetadata = tileset.findMetadata();
if (!pMetadata) {
  // No metadata available yet, probably because there's no root tile yet.
}

The findMetadata method optionally takes a Tile. If specified, the method finds the metadata associated with the external (or top-level) tileset that contains the tile.

The metadata properties mentioned above are all available on the TilesetMetadata instance via the statically-typed Cesium3DTiles classes, like Schema and GroupMetadata. These types use a lot of JsonValue instances, though, by necessity.

A new class in Cesium3DTiles called MetadataQuery makes it easier to find properties by semantic. For example:

if (pMetadata->schema && pMetadata->metadata) {
  std::optional<Cesium3DTiles::FoundMetadataProperty> found =
      Cesium3DTiles::MetadataQuery::findFirstPropertyWithSemantic(
          *pMetadata->schema,
          *pMetadata->metadata,
          "MATERIAL_VARIANTS");
  if (found) {
    const JsonValue::Array& variantsJson = found1->propertyValue.getArray();
    std::vector<std::string> variants(variantsJson.size());
    std::transform(
        variantsJson.begin(),
        variantsJson.end(),
        variants.begin(),
        [](const JsonValue& value) { return value.getStringOrDefault(""); });

    // ... do something with the variants
  }
}

The same technique can be used to access group variants as well.

The above code samples are adapted from a test in TestTilesetSelectionAlgorithm.cpp called Allows access to material variants that demonstrates (and tests) the ability to access tileset-level material variant information, based on the example in #676. @javagl would you mind taking a look to see if this meets your expectations?

Not handled here: external schemas referenced via schemaUri. Users would have to load them manually.

[javagl]

This PR builds on #711 , which builds on #710 , which builds on #706 , which builds on #705 , so I assume that I'm not supposed to really "review" that in all depth (and if I am, just drop me a note). I'm now primarily looking at this to see in how far it addresses #676 .

In terms of the core functionality of allowing access to metadata, the current state looks reasonable.

With a few caveats:

---

The Cesium3DTiles::ClassProperty and CesiumGltf::ExtensionExtStructuralMetadataClassProperty classes are equal (except for the namespace and some minor differences in the comments - when re-generating both from the latest schemas, the comments should be equal as well). The same applies to all metadata classes. I really think that there should be some Cesium3DMetadata library that contains these classes, and that is used as a dependency for Cesium3DTiles and CesiumGltf.

I could imagine that this is not considered to be in the scope of this PR-chain. And I don't know how much effort it would be to tweak the code generator accordingly. The main reason why I'm mentioning this here (in this "user-oriented" "review") is that this would be a "breaking change" for users. Not a critical one. It would mainly/only change
#include <Cesium3DTiles/Schema.h>
to
#include <Cesium3DMetadata/Schema.h>
But maybe still worth mentioning.

---

The point of

if (!pMetadata) {
  // No metadata available yet, probably because there's no root tile yet.
}

also bugged me a bit in the experiments that I did in #693 . Right now, I don't see a sensible way for clients to handle this. (while (!metadata) metadata=tryAgain()?). Two high-level approaches for handling this would be

• return a Future to the metadata
• add some callback to the Tileset class that is called when the metadata is there

with ~"hybrid" approaches like

void addCallback(Callback c) {
    this.privateMetadataFuture.whenComplete.call(c);
}

or so.

---

Not handled here: external schemas referenced via schemaUri. Users would have to load them manually.

I wonder how users should do that. Forcing users to dive into the depth of the AsyncSystem and manually (!?) calling the Cesium3DTilesReader::SchemaReader seems like a burden.

(I think that the approaches of the previous point, i.e. using a Future or Callback, could kill two birds with one stone here: Users can access the Schema in the same way, regardless of whether it is loaded with the "root tile" (i.e. with the tileset), or from an external URI. The difference could be completely hidden from the user)

---

Very narrowly focussing on #676 , users will have to...

• access the Cesium3DTilesSelection::TilesetMetadata
  • raising the quesiton of 'when it is loaded'
  • raising questions about the schemaUri
• (be aware of (!) and) use the Cesium3DTiles::MetadataQuery class
• know the Cesium3DTiles::FoundMetadataProperty
  • An aside: This does not seem to be marked as CESIUM3DTILES_API, but maybe that's an oversight on my side...
• transform the CesiumUtility::JsonValue objects into strings

I think that there could/should be a class that hides all this. This would make accessing the material variants much easier, and (as I said elsewhere) it would also hide possible changes that come from introducing something like a Cesium3DMetadata library in the future.

Note: This is also not directly in the scope of this PR. (It could rather be part of an updated state of #693 ).

But at some point, there could be classes like this....:

struct MaterialVariants {
  std::vector<std::string> tilesetMaterialVariantNames;
  std::vector<std::vector<std::string>> groupsMaterialVariantNames;
};

class MaterialVariantsUtilities {

public:
  static MaterialVariants
  fromTileset(const Cesium3DTilesSelection::Tileset &tileset) {
    MaterialVariants materialVariants;

    const Cesium3DTilesSelection::TilesetMetadata *pMetadata =
        tileset.findMetadata();
    materialVariants.tilesetMaterialVariantNames =
        MaterialVariantsUtilities::findStringPropertyValues(
            *pMetadata->schema, *pMetadata->metadata, "MATERIAL_VARIANTS");
    for (const Cesium3DTiles::GroupMetadata &group : pMetadata->groups) {

      materialVariants.groupsMaterialVariantNames.push_back(
          MaterialVariantsUtilities::findStringPropertyValues(
              *pMetadata->schema, group, "MATERIAL_VARIANTS"));
    }
    return materialVariants;
  }

  static void debugPrint(const MaterialVariants &materialVariants) {
    std::cout << "Material variants:" << std::endl;
    std::cout << "  Tileset:" << std::endl;
    const auto &t = materialVariants.tilesetMaterialVariantNames;
    for (size_t i = 0; i < t.size(); i++) {
      std::cout << "    " << t[i] << std::endl;
    }
    std::cout << "  Groups:" << std::endl;
    const auto &gs = materialVariants.groupsMaterialVariantNames;
    for (size_t i = 0; i < gs.size(); i++) {
      std::cout << "  Group " << i << ":" << std::endl;
      const auto &g = gs[i];
      for (size_t j = 0; j < g.size(); j++) {
        std::cout << "    " << g[j] << std::endl;
      }
    }
  }

private:
  static std::vector<std::string>
  findStringPropertyValues(const Cesium3DTiles::Schema &schema,
                           const Cesium3DTiles::MetadataEntity &metadataEntity,
                           const std::string &semantic) {
    std::optional<Cesium3DTiles::FoundMetadataProperty> propertiesWithSemantic =
        Cesium3DTiles::MetadataQuery::findFirstPropertyWithSemantic(
            schema, metadataEntity, semantic);

    const CesiumUtility::JsonValue::Array &propertiesJson =
        propertiesWithSemantic->propertyValue.getArray();
    std::vector<std::string> propertyValues(propertiesJson.size());
    std::transform(propertiesJson.begin(), propertiesJson.end(),
                   propertyValues.begin(),
                   [](const CesiumUtility::JsonValue &value) {
                     return value.getStringOrDefault("");
                   });
    return propertyValues;
  }
};

Meaning that users could just call

  const MaterialVariants materialVariants =
      MaterialVariantsUtilities::fromTileset(tileset);
  MaterialVariantsUtilities::debugPrint(materialVariants);

and have the information that they need.

[kring]

Thanks for taking a look, Marco!

This PR builds on #711 , which builds on #710 , which builds on #706 , which builds on #705 , so I assume that I'm not supposed to really "review" that in all depth (and if I am, just drop me a note).

Nah I broke this PR out to hopefully make it (and those parts) easier to review. I don't think those other PRs are especially interesting to you, but obviously you're more than welcome to take a look if you like. This is the important one.

I really think that there should be some Cesium3DMetadata library that contains these classes, and that is used as a dependency for Cesium3DTiles and CesiumGltf.

Yeah that'd be good! A little tricky and definitely outside the scope of this PR, though. You're right it'll be a breaking change in the future, but a relatively minor one (and we still don't stress about breaking changes in cesium-native too much).

Right now, I don't see a sensible way for clients to handle this. (while (!metadata) metadata=tryAgain()?).

Yeah it's a bit of a hassle. You have to poll. A SharedFuture (which is for practical purposes an event by another name) is a reasonable option, but it's something we can add in the future (yikes) if necessary. This "there's no metadata until the root tile is loaded" situation is not so different from the "we don't know where the tileset is so we can't zoom to it until the root tile is loaded" situation we already had, for what it's worth.

I wonder how users should do that. Forcing users to dive into the depth of the AsyncSystem and manually (!?) calling the Cesium3DTilesReader::SchemaReader seems like a burden.

Do you have a sense of how common external schemas are? I don't think I've ever seen one, so I was guessing they're rare and no one is too likely to be bothered by this. Loading an external schema should be pretty straightforward, though. Something like this should do the trick, either as a method on the Tileset or as something the user does themselves externally (this is untested and missing some error handling):

pAssetAccessor->get(*pMetadata->schemaUri).then([pMetadata](auto&& pRequest) {
  SchemaReader reader;
  auto result = reader.readFromJson(pRequest->response()->data());
  if (result) {
    pMetadata->schema = std::move(*result.value);
  }
});

If you think this will be common enough to bother with, I'm happy to add a method like this now.

An aside: This does not seem to be marked as CESIUM3DTILES_API, but maybe that's an oversight on my side...

Oops, just my oversight. I've added it. I think it's not strictly needed for types without methods, but a good idea to add anyway.

I think that there could/should be a class that hides all this. This would make accessing the material variants much easier, and (as I said elsewhere) it would also hide possible changes that come from introducing something like a Cesium3DMetadata library in the future.

I definitely agree we can make variant access easier. But I'm currently assuming this tileset-level material variant thing is specific to the company that has requested it, rather than a common thing we should make easy for everyone. If and when there's an official spec for describing tileset-level material variants, it may or may not look like it does today. So I'm hesitant to bake it into our API now. I'd rather provide some code to that one company to help them accomplish their goal for now. In fact, you've already written it in your comment! If we start to see this get traction more widely, then we can incoporate it into the API.

I may not be across the full history and status of material variants, though, so let me know if I'm thinking about it wrong.

[javagl]

Yeah it's a bit of a hassle. You have to poll. A SharedFuture (which is for practical purposes an event by another name) is a reasonable option, but it's something we can add in the future (yikes) if necessary.

I'm thinking about the convenience for the user, and the stability of the API.

Compared to things like the possible Cesium3DMetata library, this is a different category. It's a difference if the change log says:

Class Xyz was moved from Abc to Def, just do a search-and-replace on your includes and you're done

or

The return type of xyz() changed. Instead of /* 5 lines of code */ you now have to use /* 20 lines of code */

---

Sorry, I derailed a bit while writing this part 🤓 :

This "there's no metadata until the root tile is loaded" situation is not so different from the "we don't know where the tileset is so we can't zoom to it until the root tile is loaded" situation we already had

I had looked at the Tileset class in the context of all this, and was... a bit surprised to see that there is no way to detect whether the "root tile" was loaded. I'd probably have to read more in the current state of the code, but don't you think that some callback/notification mechanism could be useful and desirable in a more general sense?

As a very high-level thought, this refers to thinks like

• The idea of a PostProcessor that I saw when quickly skimming over Tileset and Tile Content Loader Refactor Overview #481
• Things like TilesetContentManager::postProcessContentInWorkerThread that already exist
• The ubiquitous if (!root) { shrug(); } pattern that can be found with https://github.com/search?q=org%3ACesiumGS%20getRootTile&type=code

Very abstractly speaking: I think that whenever there is a state change, one should consider whether it makes sense to offer the possibiltiy to send out notifications about the state change. In some contexts (mainly UIs), that's simply called "MVC". But it specifically refers to state machines that are implemented as such - e.g. the content loading process, where some onStateChanged(ContentLoading, Failed, errorCallback) or onStateChanged(ContentLoading, Loaded, postProcessingCallback) could be handy...

For the specific RootTile/Metadata question, one could argue that this can be added later, as in

// Returns nullptr when not present
Metadata* getMetadata() {....}

// Can be added later: Returns a future that resolves when it's present:
Future<Metadata> getMetadataFuture() {....}

without causing a breaking change (and even if it's breaking - we all love these 0.x.y versions, don't we? 😁 )

But often, the future-ness of some return value is "viral" for the call chain, and many places that are currently doing that if (!root) return whatever(); could benefit from a Future.

So even though the question of whether that method should return a Metadata* or a Future<Metadata> (or whether the latter can be added in the future) may not be the core of this PR, one could consider tracking this idea in a broader discussion/planning issue,...

Back to topic...

---

Do you have a sense of how common external schemas are? I don't think I've ever seen one, so I was guessing they're rare and no one is too likely to be bothered by this.

I have no idea how common this is. But I'd like to avoid issues or forum threads like "Help, my data isn't displayed properly", which involve debugging and requesting the source data, only to see that it uses a schemaUri, and a response like "Yeah, we're not handling this right now, here are possible workarounds...".

To put it that way: Many tilesets that include metadata now are dedicatedly created for testing, and there, it's simpler to just inline the schema. But when people are starting to use and create metadata in a much broader sense, then they will use it in every shape and form that is allowed by the specification (and... beyond that, but that's not our problem right now).

(One important use case for "external schemas" will actually be in the context of glTF. When you have a tileset with glTFs with complex metadata, you don't want to repeat the schema in each and every glTF. That's different from the tileset metadata, but the case that people want to share the same 3D Metadata schema between glTFs and a tileset (!) does seem like a very reasonable one).

But I'm currently assuming this tileset-level material variant thing is specific to the company that has requested it, rather than a common thing we should make easy for everyone.

That's true. And the generic access to metadata is the core of this PR.

But at the same time, I assume that we'll have to create some convenience infrastructure for accessing metadata, and semantics in particular. This refers to low-level things like the MetadataQuery class. It refers to "specification-level" questions like CesiumGS/3d-tiles#574 . And it refers to a "combination" of both, like CesiumGS/3d-tiles#643 , with the idea of treating "the set of semantics" directly as a 3D Metadata schema (!), to be machine-processable.

The findStringPropertyValues values in the class drafted above could be one tiny, tiny building block of that. Eventually, there is a "spectrum" of
std::vector<std::string>> v = MaterialVariants.getFor(tileset);
std::vector<std::string>> v = MetadataSemantics.getStringsFor(tileset, "MATERIAL_VARIANTS");
std::vector<std::string>> v = Metadata.getStrings(semanticsSchema, "tilesetSemanticsClass", "MATERIAL_VARIANTS");
...
and we have to pick the right one (always with the option of wrapping the "more convenient ones" around the "more low-level ones").

I definitely agree we can make variant access easier.

The convenience layer drafted in the MaterialVariantsUtilities may be too specific to be included in the core codebase for now. But in doubt, it should be possible to provide this to the customer/company that requested it, and be it only to say: "Just copy these 200 lines into your codebase, and you'll have that names = Magic.getThem() method that you actually need" (so that they don't have to come up with their own solutions for the "external schema" questions and such).

In that regard, I'd really like to be able to show how they could implement such a class - but for that, I'd have to show them how to handle

1. the case that the metadata is not loaded yet
2. the case that there is a schemaUri

Specifically, for the latter:

Loading an external schema should be pretty straightforward, though.

I would then include the suggested code block (as a private method) in these utilities (and see if I can figure out from where I could get the AssetAccessor that is needed for this...). This means that this utility class must already return a Future<MaterialVariants> (probably even a Future<std::optional...> ).

(That's the "viral" part of future-ness...)

[kring]

I've made some further improvements:

• Renamed findMetadata to getMetadata and added a loadMetadata that returns a SharedFuture that will only resolve once the root tile and the schemaUri (if it has one) have been loaded.
• Added a convenience function JsonValue called getArrayOfStrings that makes it a little more convenient to access the variant arrays.
• Added getRootTileAvailableEvent to Tileset (which is used in the implementation above, and possibly useful in its own right).

I don't think I can justify much more time polishing this, but let me know if you have any remaining serious concerns @javagl.

[javagl]

No further substantial comments from my side here. I tried it out with a 'material variants' example, once with an inlined and once with an external schema, and it seems to work.

[j9liu]

@kring looks good to me! Just caught a few small mistakes, but happy to merge once they're fixed + the other PRs are merged to main.

[j9liu]

Seems like this should be required so pRootTile->getChildren().front(); is valid.

Suggested change

	CHECK(pRootTile->getChildren().size() == 1);
	REQUIRE(pRootTile->getChildren().size() == 1);

[j9liu]

Suggested change

	CHECK(pRootTile->getChildren().size() == 1);
	REQUIRE(pRootTile->getChildren().size() == 1);

[j9liu]

Suggested change

	* future resolves, {@link findMetadata} can be used to synchronously obtain
	* future resolves, {@link getMetadata} can be used to synchronously obtain

[j9liu]

Rename pRenderContent to pExternalContent for clarity?

Suggested change

	const std::unique_ptr<TileExternalContent>* pRenderContent =
	std::get_if<std::unique_ptr<TileExternalContent>>(&this->_contentKind);
	if (pRenderContent) {
	return pRenderContent->get();
	}
	return nullptr;
	}
	TileExternalContent* TileContent::getExternalContent() noexcept {
	std::unique_ptr<TileExternalContent>* pRenderContent =
	std::get_if<std::unique_ptr<TileExternalContent>>(&this->_contentKind);
	if (pRenderContent) {
	return pRenderContent->get();
	}
	const std::unique_ptr<TileExternalContent>* pExternalContent =
	std::get_if<std::unique_ptr<TileExternalContent>>(&this->_contentKind);
	if (pExternalContent) {
	return pExternalContent->get();
	}
	return nullptr;
	}
	TileExternalContent* TileContent::getExternalContent() noexcept {
	std::unique_ptr<TileExternalContent>* pExternalContent =
	std::get_if<std::unique_ptr<TileExternalContent>>(&this->_contentKind);
	if (pExternalContent) {
	return pExternalContent->get();
	}

[kring]

Thanks @j9liu! This (and the other PRs it depends on) are ready for another look.

[j9liu]

Thanks @kring ! Merging now.