SDXL crashing when trying to run

[rovsiyen]

Describe the bug
SDXL is crashing immediately using latest main commit.

To Reproduce

1. git clone  https://github.com/microsoft/Olive.git
2. cd olive
3. python -m venv sdxl
4. sdxl\Scripts\activate
5. pip install e .
6. cd examples\directml\stable_diffusion_xl
7. pip install -r requirements.txt
8. python stable_diffusion_xl.py --optimize
9. python stable_diffusion_xl.py --prompt Astronaut

Expected behavior
stable_diffusion_xl.py runs and generates an image of an astronaut

Olive config
\examples\directml\stable_diffusion_xl\config.py

vae_sample_size = 1024
unet_sample_size = 128
cross_attention_dim = 2048
time_ids_size = 6

Olive logs

(sdxl) C:\Users\rovsiyen\Downloads\Olive\examples\directml\stable_diffusion_xl>python stable_diffusion_xl.py --prompt astronaut
C:\Users\rovsiyen\Downloads\Olive\sdxl\lib\site-packages\diffusers\utils\outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
Loading models into ORT session...
2024-02-06 09:30:11.6193410 [E:onnxruntime:, inference_session.cc:1935 onnxruntime::InferenceSession::Initialize::<lambda_a4a20ed4a1637918520dcc35b936125b>::operator ()] Exception during initialization: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(3059)\onnxruntime_pybind11_state.pyd!00007FF85CE8F39A: (caller: 00007FF85CE595D4) Exception(1) tid(6e74) 80004005 Unspecified error

Traceback (most recent call last):
  File "C:\Users\rovsiyen\Downloads\Olive\examples\directml\stable_diffusion_xl\stable_diffusion_xl.py", line 635, in <module>
    main()
  File "C:\Users\rovsiyen\Downloads\Olive\examples\directml\stable_diffusion_xl\stable_diffusion_xl.py", line 617, in main
    run_inference(
  File "C:\Users\rovsiyen\Downloads\Olive\examples\directml\stable_diffusion_xl\stable_diffusion_xl.py", line 255, in run_inference
    pipeline = ORTStableDiffusionXLPipeline.from_pretrained(
  File "C:\Users\rovsiyen\Downloads\Olive\sdxl\lib\site-packages\optimum\onnxruntime\modeling_ort.py", line 626, in from_pretrained
    return super().from_pretrained(
  File "C:\Users\rovsiyen\Downloads\Olive\sdxl\lib\site-packages\optimum\modeling_base.py", line 372, in from_pretrained
    return from_pretrained_method(
  File "C:\Users\rovsiyen\Downloads\Olive\sdxl\lib\site-packages\optimum\onnxruntime\modeling_diffusion.py", line 336, in _from_pretrained
    vae_decoder, text_encoder, unet, vae_encoder, text_encoder_2 = cls.load_model(
  File "C:\Users\rovsiyen\Downloads\Olive\sdxl\lib\site-packages\optimum\onnxruntime\modeling_diffusion.py", line 214, in load_model
    unet = ORTModel.load_model(unet_path, provider, session_options, provider_options)
  File "C:\Users\rovsiyen\Downloads\Olive\sdxl\lib\site-packages\optimum\onnxruntime\modeling_ort.py", line 373, in load_model
    return ort.InferenceSession(
  File "C:\Users\rovsiyen\Downloads\Olive\sdxl\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 419, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "C:\Users\rovsiyen\Downloads\Olive\sdxl\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 483, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(3059)\onnxruntime_pybind11_state.pyd!00007FF85CE8F39A: (caller: 00007FF85CE595D4) Exception(1) tid(6e74) 80004005 Unspecified error

Other information

• OS: Windows 11
• Olive version: main
• ONNXRuntime package and version: onnxruntime-directml==1.17.0
• GPUs: Arc770, Nvidia3060, Nvidia4060, Nvidia4080, RX7600

Additional context
Tried on multiple platforms, issue is reproducible on all systems listed above.

[xiaoyu-work]

@jstoecker @PatriceVignola can you please take a look at this?

[PatriceVignola]

@rovsiyen We have a PR to fix the issue: microsoft/onnxruntime#19415 It will be merged into main soon and will be part of a 1.17.1 patch release.

In the meantime you can either

1. Checkout the commit of this PR locally and build from source
2. Checkout the last good commit (1bab98988b4e7b6d33be0e672fce361ccbb1d397) and build from source
3. Fall back to the 1.16.3 pypi release.

[xiaoyu-work]

Closing as no response. Please reopen this if you have more questions.