Spatial subdivision for USD

[pjcozzi]

From #519:

3D Tiles has the opportunity to broadly solve spatial subdivision and metadata interoperability, e.g.,

1. Spatial subdivision with USD as the tile payloads
   ...

I think this is a big enough topic to separate into this issue for discussion.

3D Tiles explicit and implicit tiling could be decoupled from 3D Tiles to be able to represent the spatial subdivision of any content: glTF and USD, for example, for models; and pure metadata for things like vector fields, distance fields, etc. Spatial subdivision may be used for streaming at runtime to a client but also for accelerating server-side computations or distributing computation across a cloud.

USD is used in NVIDIA Omniverse and UDZ is used in the Apple ecosystem. Ignoring metadata, it seems straightforward to create a 3DTILES_content_USD extension (#519 (comment)) to allow 3D Tiles to be a spatial container for USD (ditto for USDZ), which may be the answer or part of the answer.

We should also take a closer look at USD to see if there is value is embedding 3D Tiles explicit and/or implicit tiling inside USD (the analog of #553 for glTF).

This is not urgent for 3D Tiles Next to go from an open specification to an open standard, but it would be great to have initial ideas on the path to cleanly decouple spatial subdivision from 3D Tiles for general use.

[ptrgags]

If we do create a 3DTILES_content_USD, we want to think about MIME types, see #388

[pjcozzi]

@lilleyse could you please share some thoughts on how 3D Tiles implicit and/or explicit tiling could be used for USD? Consider how an end-to-end author (USD) and publish (glTF) workflow could be interoperable.

SideFX has an awesome USD Basics tutorial.

Two broad directions:

1. USD as a 3D Tiles tile content, e.g., 3DTILES_content_USD
2. 3D Tiles as a spatial data structure inside USD

Metadata interoperability is important to both topics.

For (2), USD supports

• Compositing Layers by referencing other .usd files to extend the scene graph / node hierarchy / tree. AFAICT this can be used spatially like a traditional BVH but it can also be used via sub-layers for shading effects. Seems like 3D Tiles could fit here.
• Payloads are used for out-of-code loading and may also be a touch point for 3D Tiles
• A usdz file can contain multiple layers / USD files and may be inspiration for an archive for 3D Tiles, Package tileset in a single file #89, but I don't think is in scope for this issue.

[pjcozzi]

@lilleyse as part of this research, seems like it is worth checking out the open-source USD library and seeing how Cesium Native could be combined, e.g., if tile content payloads are abstracted away in Cesium Native for a plugin point for glTF or USD content, and how metadata would be accessed. I believe the USD library API is widely used and end users would want to access it, not an abstraction layer above it, but I am noob here.

[lilleyse]

Here's a few workflows I've been thinking about:

• Export workflow: USD is exported as 3D Tiles with no common spatial hierarchy between the two. The layers are "baked" into a single flat scene and the scene is tiled. Per-prim (per-feature) metadata is preserved but now represented as 3D Metadata.
• Edit/play workflow: 3D Tiles provides the top-level spatial hierarchy. Each tile has USD content and glTF content. When a user makes changes the USD is modified, and when they are done making changes the USD is converted into an optimized glTF (batching, compression, etc). This would be like going between edit mode and play mode in a game engine but could happen in real time too. The 3DTILES_multiple_contents extension could be used here.
• 3D Tiles inside USD: 3D Tiles would provide hierarchical level of detail for large base layers (terrain, cities, etc) within a USD scene. May be implemented as on-the-fly transcode from tiles to xforms, variants, payloads, and camera triggers.
• 3D Tiles drives USD state: 3D Tiles as a spatial hierarchy of behavior where tile payload is a command list for the USD API such as switching variants or toggling visibility. Or could be implemented as user-defined callbacks.