Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for USD and USDZ formats #14219

Closed
HenryB96 opened this issue Jun 4, 2018 · 73 comments
Closed

Add support for USD and USDZ formats #14219

HenryB96 opened this issue Jun 4, 2018 · 73 comments

Comments

@HenryB96
Copy link

HenryB96 commented Jun 4, 2018

It would be great to get Three.js support for the Universal Scene Description (USD) file format and the upcoming USDZ format announced at WWDC today.

USDZ specification document: https://graphics.pixar.com/usd/files/USDZFileFormatSpecification.pdf (I assume that more details will come out in the coming months for this format)

@Mugen87
Copy link
Collaborator

Mugen87 commented Jun 5, 2018

It's a valid feature request but I personally don't understand Apple/Pixar. Why are they introducing a new file format and not just using glTF? Can somebody explain what USD /USDZ does better than glTF?

@Usnul
Copy link
Contributor

Usnul commented Jun 6, 2018

USD documentation: http://graphics.pixar.com/usd/docs/index.html

It's a valid feature request but I personally don't understand Apple/Pixar. Why are they introducing a new file format and not just using glTF? Can somebody explain what USD/USDZ does better than glTF?

I agree, part of it is politics me thinks. However, that's a separate discussion :)

@donmccurdy
Copy link
Collaborator

donmccurdy commented Jun 6, 2018

USD has been around for a couple years — it's commonly used in film, and not an obvious choice for runtime transmission. As far as I can tell USDZ is a wrapper around that. Apple hasn't given much detail at this point; maybe they have some plan to make it more runtime-appropriate, or maybe the messaging at WWDC was just unclear with respect to how they're actually using it. In any case I think we probably need to wait for more information.

If someone is interested in contributing a THREE.USDZLoader, then by all means. 🙂 But all of the documentation I've found so far suggests the format is very closely tied to the official (C++/Python) libraries for parsing it, similar to FBX and the FBX SDK. So unless those can be made web-friendly with WASM, implementing USDZ support in three.js and keeping up with changes in the format over time looks like a very time-consuming process, without the benefit of those libraries.

@mrdoob mrdoob mentioned this issue Jun 16, 2018
42 tasks
@richardmonette
Copy link
Contributor

Out of curiosity, I tried opening the "Retro TV" file from https://developer.apple.com/arkit/gallery/ as if it was a simple zip. I 'unarchived' the .usdz by renaming it's extension to .zip, and then having my OS do its usual open procedure. Presumably, something similar could be achieved with https://gildas-lormeau.github.io/zip.js/

This is what I see when I 'unarchive' the usdz:

screen shot 2018-07-04 at 2 17 46 pm

So, the textures are just PNGs, which are probably referenced from within that .usdc, which is the next issue. According to https://graphics.pixar.com/usd/docs/Converting-Between-Layer-Formats.html, ".usdc files – binary format" which is in Crate (see Crate File Format in https://graphics.pixar.com/usd/docs/USD-Glossary.html) format.

There is also .usda, which is ASCII encoded, which would presumably be easier to write a loader around.

As a rough outline of some to-dos for this, minimally a Loader would be looking at extracting a Zip file in JS, and then also understanding the .usda (and potentially binary Crate .usdc as well.)

@TimvanScherpenzeel
Copy link

Hi @richardmonette, I've done some research on USDZ and the USDA format and have written a proof-of-concept glTF to USDZ converter. Perhaps it is useful for your research: https://github.com/timvanscherpenzeel/gltf-to-usdz. You can see a demo here: https://twitter.com/domenicopanacea/status/1011292393801420800 or try it out yourself here: https://timvanscherpenzeel.github.io/gltf-to-usdz/ (requires iOS 12).

In short: geometry gets converted to OBJ, the metallic-roughness texture is split by channel and textures are Y-flipped. From this a USDA file is constructed dynamically and fed into usdz_converter to construct the final usdz.

I know some people are looking into writing a direct converter to USDZ but not a whole lot is known about the USDC format at this point in time (also in terms of decoding).

@richardmonette
Copy link
Contributor

Hey @TimvanScherpenzeel, thanks for your message and work on this!

Have you considered using the USD Python API (https://graphics.pixar.com/usd/docs/Hello-World---Creating-Your-First-USD-Stage.html)? I wondered if it might be possible to skip the .obj stage and, using Python, read the glTF and go straight into .usda/c that way?

@TimvanScherpenzeel
Copy link

I've had a lot of issues setting up USD and have not yet taken the time to fix the issue (I think it has something to do with having multiple versions of Python installed and during compilation it links to the wrong Python).

You could indeed write the OBJ file directly into the USDA file (that could actually be a good next step in my proof of concept). I know the file structure of the AR Gallery examples because a contributor to my project send me the USDA files after having the USDC files decoded it with usdcat.

@PixelPartner
Copy link

The whole point of Apple providing a USDZ QuickLook is to totally bypass using WebGL, DOM and Javascript and directly use the scene with ARkit2.0.
It's just a milestone to an rOS based on Apple/Pixar tech.
When they later add interactivity/scriptability, even less Web-UI will be needed.

Pixar USD, using highly efficient OpenSubDiv Patches for geometry, ported to mobile Metal will soon outperform HighEnd PC-VR WebVR.
Not as a file format, but as a whole Experience-Making/Presenting/(-Interaction) pipeline.

So IMHO it's more important to export USDZ from AR/VR Editors and for Android and Windows to catch the USD train, than to add an importer to the WebGL2 pipeline.

An important issue is that WebXR would have to also support SubDiv patches instead of legacy polymeshes to fully support USD as used by Apple.

@FreakTheMighty
Copy link

@PixelPartner you sorta lost me.

It seems to me that unless a loader is built natively into browsers, it would be helpful to build one in JavaScript..

I’m sure quick look is highly optimized, but what is the path forward for creating interactive content using USDZ assets in a browser?

@looeee
Copy link
Collaborator

looeee commented Sep 11, 2019

It's a valid feature request but I personally don't understand Apple/Pixar. Why are they introducing a new file format and not just using glTF? Can somebody explain what USD /USDZ does better than glTF?

I had a chat with someone working on product display software recently, they export to glTF and USDZ so I asked him what he thought about this. His take in a nutshell: it's so that Apple can continue to claim to be the high-end luxury choice, with an exclusive 3D format that only works on their platform via ARKit2.0/SceneKit. Adopting the standard glTF format would run contrary to their goals here, they want to be able to claim that their format is better and allows for higher quality results due to PBR material properties that glTF doesn't (yet?) support.

#16977, and perhaps #14051, are likely needed to get three.js up to the same rendering standards as Apple's offerings, but I'm not sure how the current state or future plans for glTF relate to this.

In any case, if for no other reasons than to subvert Apple's nefarious goals here, I think we'll need to support USDZ at some point.

@donmccurdy
Copy link
Collaborator

donmccurdy commented Sep 11, 2019

it's so that Apple can continue to claim to be the high-end luxury choice, with an exclusive 3D format that only works on their platform...

I'm willing to give more benefit of the doubt here – they have different goals for a "runtime" format than we (threejs + web devs) do, and are able to make different choices as a result. In particular, bundling the ~15MB USD runtime into iOS means they don't care that loading USDZ without a native runtime is prohibitively difficult, and they seem to be OK with not providing the web application any access to the model, but just opening the model as-is in a native app instead.

On "high-end luxury", Apple's iOS QuickLook viewer supports metallic/roughness PBR materials, and the material model is largely equivalent to what glTF 2.0 supports today. The additional material features planned for glTF 2.0 in the future, and described in #16977, would go beyond what Apple's USDZ viewers support now.

I mention "Apple's USDZ viewers" because that is, de facto, what people seem to mean by USDZ. But the USDZ format, technically, is just a zipped USD file, and could contain pretty much anything from a common material model to a completely custom shader. Apple's viewers support a small subset of USD, and that subset isn't well defined, but the best summary you're likely to find is here: https://github.com/google/usd_from_gltf#compatibility.

In terms of rendering fidelity, I'd point out that model-viewer displays glTF models with threejs and provides the means to open a USDZ version in iOS Quick Look, and the visual results are quite similar. This page provides some direct comparisons, although unfortunately it is down right now.

In any case, if for no other reasons than to subvert Apple's nefarious goals here, I think we'll need to support USDZ at some point.

See #14219 (comment).

@garyo
Copy link
Contributor

garyo commented Feb 14, 2020

I just want to mention here that there are reasons for supporting USD beyond Apple "claiming to be a luxury choice". @Mugen87 asks "Can somebody explain what USD/USDZ does better than glTF?"
A few things that I care about:

  • HDR/float textures
  • Environment maps (UsdLuxDomeLight)
  • Area and sphere lights (UsdLuxRectLight etc.)
  • Shader materials (OSL/MDL) with "preview surface" for WebGL

My use case is loading the same scene description into WebGL and into a high-end rendering pipeline - understanding the looks will differ of course and some substitution will take place.

@themightyatom
Copy link

themightyatom commented Apr 14, 2020

@garyo lighting and environment are handled by the AR viewer, in an attempt to simulate the actual environment.
This video gives you an idea of what maps they support (albedo, metallic, roughness, normal, ambient occlusion, emissive)
https://developer.apple.com/videos/play/wwdc2018/603/

What I'd like to see is an USDA exporter from three.js, so you can save models you have created/ configurered in a three project, for viewing in AR.
"model-view" includes a tag for specifying a USDZ model alternative, for when viewed on iOS.
https://modelviewer.dev/

It really is a shame Apple went for a format no-one else uses, and isn't natively supported anywhere (beta exporter is available for Blender). So you have no choice but manually convert all your existing models. The fact that you have to include your textures is also a pain if you use the same texture over an entire range of models. Surely they could have external references?
It's really a bit of an hodgepodge format, that only utilises geometry and mapping from the Pixar specs, and zips them together with your image files.
glTF/glb would have been much better and we would only need ONE model for everywhere...
In the realm of e-commerce, which is what they are aiming at, no customer will want a web experience that ONLY works for iOS. As usual we'll have to do all the monkey work and maintenance.

@makc
Copy link
Contributor

makc commented Apr 14, 2020

@garyo

My use case is loading the same scene description into WebGL and into a high-end rendering pipeline

wait, is there a way to load usd(z) in webgl?

@PixelPartner
Copy link

PixelPartner commented Apr 14, 2020 via email

@donmccurdy
Copy link
Collaborator

donmccurdy commented Apr 14, 2020

I'm not aware of any web-compatible parser for USD or USDZ. Rendering in WebGL or another API isn't a problem, it's parsing the file's content that is complex — that task is best handled by the official USD runtime, which has dependencies including OpenSubdiv and MaterialX.

Any developer can now use the Pixar use tools and libs (C++/python 2.6/python 3.x) on any platform to write USD files..

Being able to read USD files on any platform would be nice. 🙂 For technical reasons I do not think a functional three.js implementation is likely right now. If someone is blessed with the spare time, my suggestion would be to work on implementing OpenSubdiv (w/ WASM?) or MaterialX (w/ THREE.NodeMaterial?), as those would be useful in their own right, even without a USD parser.

@garyo
Copy link
Contributor

garyo commented Apr 16, 2020

@garyo

My use case is loading the same scene description into WebGL and into a high-end rendering pipeline

wait, is there a way to load usd(z) in webgl?

No, that's what I'm looking for. I have scenes (object, lights, materials, camera, animation, etc. - the whole thing) and I want to be able to load the same scene, in whatever format, into WebGL (via three.js) for preview, and something like Blender for final renders. glTF is my best option for now, but it doesn't support the things I mention above (area lights etc.) I understand the glTF folks are reluctant to add more features because they're targeting last-mile to devices rather than true scene transfer.
Just looking for options.

@themightyatom
Copy link

@garyo Sounds like Collada might fit the bill? I use an edited version of the collada exporter from three.js to take assets into Blender for renders.

@garyo
Copy link
Contributor

garyo commented Apr 17, 2020

@themightyatom I looked into Collada a couple of years ago -- may have to revisit it. Thanks. (At the time I was convinced by arguments such as the Godot one, no PBR support, etc -- that may have changed.)

@themightyatom
Copy link

@garyo Collada was designed for transferring assets around between game developers all working on different platforms. It's not a viable delivery format, but it does what it's meant for very well :)
I was lucky enough to spend a week in Italy with one of it's creators, Rene Arnaud. It's written "from the GPU's perspective" and is generally the first format to support changes in spec. (OpenGL, DirectX, etc.).

@ronpadz
Copy link

ronpadz commented Jul 1, 2020

I think what would be super valuable is to integrate https://github.com/google/usd_from_gltf as an exporter. Much less important to me to be able to open USD.

@donmccurdy
Copy link
Collaborator

donmccurdy commented Jul 1, 2020

To do anything useful with USDZ (load or export) you are going to need the USD SDK. USD from glTF requires the USD SDK as a dependency, but the SDK isn't supported on the web. Apple gets around this by leaving the web browser and taking you into their native AR Quick Look application, but that's hardly a solution for the web in general.

@c4b4d4
Copy link

c4b4d4 commented Jan 14, 2021

Has anyone figured out something yet? Or it's literally impossible at the moment? I'm looking forward to use for the following pipeline:

  1. Create a scene with Three.JS
  2. Export it to GLTF for non-iOS and export it to USDZ to enable it for iOS devices (this is where I see no possibility)
  3. Load it to <model-viewer>

It works great on desktop and non-iOS devices. And I cannot rely on servers converting the files, because it's all done client-side. User generates the scene on the web that needs to be displayed in AR.

@mrdoob
Copy link
Owner

mrdoob commented Jan 14, 2021

It works great on desktop and non-iOS devices.

What desktop?

@c4b4d4
Copy link

c4b4d4 commented Jan 14, 2021

It works great on desktop and non-iOS devices.

What desktop?

MacOS+Chrome (Obvs AR not being used) Talking about <model-viewer>, which forces USDZ for iOS devices.

@darkvertex
Copy link

WASM should work in iOS Mobile Safari since iOS 11 (says caniuse.com.)

@ponahoum
Copy link

@PixelPartner it runs on Mac Safari because latest Safari versions now support SharedArrayBuffer / Atomics objects, which this implementation uses.

@jeffscottward @darkvertex Doesn't seem to work on my iphone with iOS 15.5 either and it's pretty hard to debug as i don't have a mac. I'll take a look when i have time. WASM is of course supported on iOS alongside with WebGL2 in latests versions so this is not the cause of the issue.

@ajmas
Copy link

ajmas commented Jun 24, 2022

On iOS there are some experimental settings that might need enabling, but I also would need to investigate. I don’t currently have access to my computer or a USD file. If anyone has one, I could experiment while I am travelling.

@netgfx
Copy link

netgfx commented Jun 24, 2022

@ajmas I have tried this model:
Match_Chair_White_-_100102.zip
Direct file: https://qsxfdqhsuyovskknxkaj.supabase.co/storage/v1/object/public/threed/models/Match_Chair_White_-_100102.usdz?t=2022-06-24T08%3A07%3A32.406Z

And on web, it seems to be working fine. Also, https://sketchfab.com/ offers auto-generated USDZ files with each model (free and paid ones), so that is a good source of models too.

@MAG-AdrianMeredith
Copy link

iOs is throwing a RangeError: Out of memory. We've seen this error before, its a bug in safari where the new thread support requires a little bit extra memory on startup than it thinks it does. It should load fine if you increase the initial memory to the wasm module

@MAG-AdrianMeredith
Copy link

image

@MAG-AdrianMeredith
Copy link

We fixed it by tweaking the compile flags in emscripten like this (your values may vary!)
"-sINITIAL_MEMORY=33554432",
"-sMAXIMUM_MEMORY=1073741824",

@ponahoum
Copy link

ponahoum commented Jun 24, 2022

@MAG-AdrianMeredith I can confirm that on iOS the error is RangeError: Out of memory. For some reason the WASM module was trying to assign more memory that it could. I did put a memory limit for iOS devices on three-usdz-loader latest version and it now works like a charm on my iPhone 12 Pro with iOS 15.5.

@netgfx i tried with your model and it works pretty well now !

screenshot_chair

@MAG-AdrianMeredith
Copy link

MAG-AdrianMeredith commented Jun 25, 2022 via email

@expenses
Copy link

expenses commented Aug 13, 2022

here is another sample file (2.6mb zipped, created with object capture api): saeukkang.usdz.zip

I looked into this file and the format is very strange. The points and texture coord arrays have different lengths and have individual index buffers.

Edit: same with this one:

If anyone wants to have a closer look, here is a sample USDZ file created with the ObjectCapture API. ObjectCapture_sample.zip

@ponahoum
Copy link

@disjukr @expenses both open well it seems with three-usdz-loader

saeukkang

combined_reduced

@mrdoob
Copy link
Owner

mrdoob commented Aug 31, 2022

Here's a very naive first version of the USDZLoader: #24568

I've only tried it with the models shared in this thread. It only supports usdz files that use usda internally.

The next steps are to gather as many sample files as we can (using usd/usda) and add as many features as possible.

@jeffscottward
Copy link

jeffscottward commented Aug 31, 2022

I just console.log'd against your sample file and it loaded great 👏 .

I generated my own .usdz file from the "Kaedim" AI service that turns 2D photos into 3D, which I THEN took into Reality Composer to create a facial target to lock onto and added some textures and normals.

No dice as expected with this loader. Just got silent fail of an empty object.

After getting USDTools working on Apple Silicon (https://developer.apple.com/forums/thread/702189?answerId=711711022#711711022) - I unzipped the .usdz file to get the texture files and a .usdc, then used usdcat to convert to .usda.

The .usda was successfully output but did not help in rendering a model when usdzconverting to .usdz from that. Same empty object.

I don't know if this at all helps but thought I would run it through. Here is the .usdz file in question.
https://cloudflare-ipfs.com/ipfs/QmcgmRXJZmeY51mvLikhEm3yo8oxrz15WJhBmQUyU8yTmi/ssj2-face.usdz
(Github wouldn't allow .usdz file type - had to go with slow IPFS option, give it a minute)

PS: The model is supposed to be like wearing a yellow Super Saiyan wig.
iOS QuickLook should automatically lock it onto your face.

@PixelPartner
Copy link

This animated flag pole uses SkelAnim to move the flag. If you need an example for that
https://usdzshare.com/?ug-gallery=photo-detail&photo_id=6557

@PixelPartner
Copy link

PixelPartner commented Sep 1, 2022

This seems to be the new way Apple accepts Object placement since the latest update of RealityConverter.
I'd like to point out how fine tuned pivot placement is now possible and how this differs from vanilla USD.
IMHO this was introduced to handle all kind of CAD export needs.

float3 xformOp:rotateXYZ:geometricRotation = (0, 0, 0)
float3 xformop:rotateXYZ:postRotation = (0,0,0)
float3 xform0p:rotateXYZ:preRotation = (0,0,0)
float3 xformOp:rotateXYZ:rotateXYZ = (0,0,0)
float3 xformOp:scale:geometricScaling = (1,1,1)
float3 xformOp:scale:scale =(1,1,1)
float3 xformOp:translate:geometricTranslation=(0,0,0)
float3 xformOp:translate:rotationOffset=(0,0,0)
float3 xformOp:translate:rotationPivot = (0,0,0)
float3 xform0p:translate:scalingOffset = (0,0,0)
float3 xformOp:translate:scalingpivot=(0,0,0)
float3 xformOp:translate:translate=(0,0,0)
uniform token[] xformOpOrder=[ "xformOp:translate:translate", "xformOp:translate:rotationoffset", "xformOp:translate:rotationpivot", "xformOp:rotateXYZ:preRotation", "xformOp:rotateXYZ:rotateXYZ", "xformOp:rotateXYZ:postRotation", "!invert!xformOp:translate:rotationPivot", "xformOp:translate:scalingOffset", '"xformOp:translate:scalingpivot", "xformOp:scale:scale", "!invert!xformOp:translate:scalingPivot", "xformOp: translate:geometricTranslation",
"xformOp:rotateXYZ:geometricRotation",
"xformOp:scale:geometricScaling" ]

@mrdoob
Copy link
Owner

mrdoob commented Sep 1, 2022

@jeffscottward

I don't know if this at all helps but thought I would run it through. Here is the .usdz file in question.
https://cloudflare-ipfs.com/ipfs/QmcgmRXJZmeY51mvLikhEm3yo8oxrz15WJhBmQUyU8yTmi/ssj2-face.usdz
(Github wouldn't allow .usdz file type - had to go with slow IPFS option, give it a minute)

That link doesn't work for me... I think you should be able to zip the usdz file and attach the zip to your post here.

@mrdoob
Copy link
Owner

mrdoob commented Sep 1, 2022

Attaching the file here in case those links die: ssj2-face.usdz.zip

@mrdoob
Copy link
Owner

mrdoob commented Sep 1, 2022

That file uses a usdc internally. We can't parse these yet.

Screen Shot 2022-09-01 at 4 14 12 PM

@jeffscottward
Copy link

jeffscottward commented Sep 2, 2022 via email

@mrdoob
Copy link
Owner

mrdoob commented Sep 2, 2022

But it doesn't seem like it was actually converted to usda? 🤔

Repository owner deleted a comment from jeffscottward Sep 2, 2022
@jeffscottward
Copy link

jeffscottward commented Sep 2, 2022 via email

@ghost
Copy link

ghost commented Aug 8, 2023

it's so that Apple can continue to claim to be the high-end luxury choice, with an exclusive 3D format that only works on their platform...

I'm willing to give more benefit of the doubt here – they have different goals for a "runtime" format than we (threejs + web devs) do, and are able to make different choices as a result. In particular, bundling the ~15MB USD runtime into iOS means they don't care that loading USDZ without a native runtime is prohibitively difficult, and they seem to be OK with not providing the web application any access to the model, but just opening the model as-is in a native app instead.

On "high-end luxury", Apple's iOS QuickLook viewer supports metallic/roughness PBR materials, and the material model is largely equivalent to what glTF 2.0 supports today. The additional material features planned for glTF 2.0 in the future, and described in #16977, would go beyond what Apple's USDZ viewers support now.

I mention "Apple's USDZ viewers" because that is, de facto, what people seem to mean by USDZ. But the USDZ format, technically, is just a zipped USD file, and could contain pretty much anything from a common material model to a completely custom shader. Apple's viewers support a small subset of USD, and that subset isn't well defined, but the best summary you're likely to find is here: https://github.com/google/usd_from_gltf#compatibility.

In terms of rendering fidelity, I'd point out that model-viewer displays glTF models with threejs and provides the means to open a USDZ version in iOS Quick Look, and the visual results are quite similar. This page provides some direct comparisons, although unfortunately it is down right now.

In any case, if for no other reasons than to subvert Apple's nefarious goals here, I think we'll need to support USDZ at some point.

See #14219 (comment).

@jeffscottward
Copy link

Worth mentioning a new joint effort to further the ecosystem with this new alliance.

https://aousd.org/

Pixar Adobe Apple Autodesk Nvidia Epic Unity Cesium and others

@ShaddyDC
Copy link

ShaddyDC commented Sep 4, 2023

I recently came across tinyusdz which claims to support compilation to WASM. I should note that I've used neither three.js nor the library, but maybe this is useful for people looking into this.

@Mugen87
Copy link
Collaborator

Mugen87 commented Feb 13, 2024

Since three.js offers USDZLoader and USDZExporter for quite a while now, this issue can be closed.

Please file module specific feature requests or bug reports in new issues.

@yosun
Copy link

yosun commented Jul 9, 2024

Since three.js offers USDZLoader and USDZExporter for quite a while now, this issue can be closed.

Please file module specific feature requests or bug reports in new issues.

yep modelviewer.dev has always had usdz support ("free quicklook" no iOS/VIsionOS) via three.js even though they are advocating for glb/gltf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests