[Bug]: Stable Diffusion XL not found at path – dev branch

[kopyl]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

I launched Web UI as python webui.py --port 3000 --api --xformers --enable-insecure-extension-access --ui-debug-mode and got this:

Warning: Stable Diffusion XL not found at path /runpod-volume/stable-diffusion-webui/repositories/generative-models/sgm
Traceback (most recent call last):
  File "/runpod-volume/stable-diffusion-webui/webui.py", line 61, in <module>
    from modules import shared, sd_samplers, upscaler, extensions, localization, ui_tempdir, ui_extra_networks, config_states
  File "/runpod-volume/stable-diffusion-webui/modules/sd_samplers.py", line 1, in <module>
    from modules import sd_samplers_compvis, sd_samplers_kdiffusion, shared
  File "/runpod-volume/stable-diffusion-webui/modules/sd_samplers_compvis.py", line 9, in <module>
    from modules import sd_samplers_common, prompt_parser, shared
  File "/runpod-volume/stable-diffusion-webui/modules/sd_samplers_common.py", line 5, in <module>
    from modules import devices, processing, images, sd_vae_approx, sd_samplers, sd_vae_taesd
  File "/runpod-volume/stable-diffusion-webui/modules/processing.py", line 16, in <module>
    import modules.sd_hijack
  File "/runpod-volume/stable-diffusion-webui/modules/sd_hijack.py", line 5, in <module>
    import modules.textual_inversion.textual_inversion
  File "/runpod-volume/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 16, in <module>
    from modules import shared, devices, sd_hijack, processing, sd_models, images, sd_samplers, sd_hijack_checkpoint, errors, hashes
  File "/runpod-volume/stable-diffusion-webui/modules/sd_models.py", line 17, in <module>
    from modules import paths, shared, modelloader, devices, script_callbacks, sd_vae, sd_disable_initialization, errors, hashes, sd_models_config, sd_unet, sd_models_xl
  File "/runpod-volume/stable-diffusion-webui/modules/sd_models_config.py", line 9, in <module>
    sd_xl_repo_configs_path = os.path.join(paths.paths['Stable Diffusion XL'], "configs", "inference")
KeyError: 'Stable Diffusion XL'

I have Python 3.10.6

Steps to reproduce the problem

1. Go to ....
2. Press ....
3. ...

What should have happened?

...

Version or Commit where the problem happens

ec83db8

What Python version are you running on ?

Python 3.10.x

What platforms do you use to access the UI ?

Linux

What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

Cross attention optimization

xformers

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

`python webui.py --port 3000 --api --xformers --enable-insecure-extension-access --ui-debug-mode`

List of extensions

Controlnet

Console logs

Warning: Stable Diffusion XL not found at path /runpod-volume/stable-diffusion-webui/repositories/generative-models/sgm
Traceback (most recent call last):
  File "/runpod-volume/stable-diffusion-webui/webui.py", line 61, in <module>
    from modules import shared, sd_samplers, upscaler, extensions, localization, ui_tempdir, ui_extra_networks, config_states
  File "/runpod-volume/stable-diffusion-webui/modules/sd_samplers.py", line 1, in <module>
    from modules import sd_samplers_compvis, sd_samplers_kdiffusion, shared
  File "/runpod-volume/stable-diffusion-webui/modules/sd_samplers_compvis.py", line 9, in <module>
    from modules import sd_samplers_common, prompt_parser, shared
  File "/runpod-volume/stable-diffusion-webui/modules/sd_samplers_common.py", line 5, in <module>
    from modules import devices, processing, images, sd_vae_approx, sd_samplers, sd_vae_taesd
  File "/runpod-volume/stable-diffusion-webui/modules/processing.py", line 16, in <module>
    import modules.sd_hijack
  File "/runpod-volume/stable-diffusion-webui/modules/sd_hijack.py", line 5, in <module>
    import modules.textual_inversion.textual_inversion
  File "/runpod-volume/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 16, in <module>
    from modules import shared, devices, sd_hijack, processing, sd_models, images, sd_samplers, sd_hijack_checkpoint, errors, hashes
  File "/runpod-volume/stable-diffusion-webui/modules/sd_models.py", line 17, in <module>
    from modules import paths, shared, modelloader, devices, script_callbacks, sd_vae, sd_disable_initialization, errors, hashes, sd_models_config, sd_unet, sd_models_xl
  File "/runpod-volume/stable-diffusion-webui/modules/sd_models_config.py", line 9, in <module>
    sd_xl_repo_configs_path = os.path.join(paths.paths['Stable Diffusion XL'], "configs", "inference")
KeyError: 'Stable Diffusion XL'

### Additional information

_No response_

[wangqyqq]

cd repositories
git clone https://github.com/Stability-AI/generative-models.git

Then you can use SDXL

[kopyl]

@ljbq thank you, i probably used the wrong directory. Will check it and then close this issue.

[akx]

Or run launch.py instead of webui.py so it downloads that for you.

[kopyl]

@akx i was wrong here, I did not use the correct web-ui directory, sorry. The one where i launched the installer works fine :)

[metter]

I am having the same issue, but during docker build.
I included the above repo before the caching of the model for runpod serverless, but it fails the cache with exact the same error message.

in Dockerfile

WORKDIR /repositories
RUN git clone https://github.com/Stability-AI/generative-models.git

COPY builder/cache.py /stable-diffusion-webui/cache.py
RUN cd /stable-diffusion-webui && python cache.py --use-cpu=all --ckpt /model.safetensors

error message:

=> ERROR [stage-1 16/27] RUN cd /stable-diffusion-webui && python cache.py --use-cpu=all --ckpt /model.safetensors                                            5.3s
------                                                                                                                                                              
 > [stage-1 16/27] RUN cd /stable-diffusion-webui && python cache.py --use-cpu=all --ckpt /model.safetensors:                                                       
#29 3.944 Warning: Stable Diffusion XL not found at path /stable-diffusion-webui/repositories/generative-models/sgm                                                 
#29 4.405 No module 'xformers'. Proceeding without it.                                                                                                              
#29 4.411 Warning: caught exception 'Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx', memory monitor disabled                                                                                                                    
#29 4.545 Traceback (most recent call last):
#29 4.545   File "/stable-diffusion-webui/cache.py", line 1, in <module>
#29 4.545     from webui import initialize
#29 4.545   File "/stable-diffusion-webui/webui.py", line 61, in <module>
#29 4.545     from modules import shared, sd_samplers, upscaler, extensions, localization, ui_tempdir, ui_extra_networks, config_states
#29 4.545   File "/stable-diffusion-webui/modules/sd_samplers.py", line 1, in <module>
#29 4.545     from modules import sd_samplers_compvis, sd_samplers_kdiffusion, shared
#29 4.545   File "/stable-diffusion-webui/modules/sd_samplers_compvis.py", line 9, in <module>
#29 4.546     from modules import sd_samplers_common, prompt_parser, shared
#29 4.546   File "/stable-diffusion-webui/modules/sd_samplers_common.py", line 5, in <module>
#29 4.546     from modules import devices, processing, images, sd_vae_approx, sd_samplers, sd_vae_taesd
#29 4.546   File "/stable-diffusion-webui/modules/processing.py", line 16, in <module>
#29 4.546     import modules.sd_hijack
#29 4.546   File "/stable-diffusion-webui/modules/sd_hijack.py", line 5, in <module>
#29 4.546     import modules.textual_inversion.textual_inversion
#29 4.546   File "/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 16, in <module>
#29 4.546     from modules import shared, devices, sd_hijack, processing, sd_models, images, sd_samplers, sd_hijack_checkpoint, errors, hashes
#29 4.546   File "/stable-diffusion-webui/modules/sd_models.py", line 17, in <module>
#29 4.546     from modules import paths, shared, modelloader, devices, script_callbacks, sd_vae, sd_disable_initialization, errors, hashes, sd_models_config, sd_unet, sd_models_xl
#29 4.546   File "/stable-diffusion-webui/modules/sd_models_config.py", line 9, in <module>
#29 4.546     sd_xl_repo_configs_path = os.path.join(paths.paths['Stable Diffusion XL'], "configs", "inference")
#29 4.546 KeyError: 'Stable Diffusion XL'
------
executor failed running [/bin/bash -o pipefail -c cd /stable-diffusion-webui && python cache.py --use-cpu=all --ckpt /model.safetensors]: exit code: 1

any idea why this still happens?

[akx]

@metter What dockerfile is that? I don't think there is one in this repo.

[metter]

@metter What dockerfile is that? I don't think there is one in this repo.

I am using a fork of this: https://github.com/runpod-workers/worker-a1111
I tried to update it to a1111 1.5.1 but ran into the above problem.
But I have figured it out since:
The problem was that I only cloned generative models in the respective folder, but didn't install it.
Adding this to my Dockerfile solved it:

# Install generative models
WORKDIR /stable-diffusion-webui/repositories/generative-models

# Install required packages from pypi inside the virtual environment
RUN python3 -m venv .pt2
RUN . .pt2/bin/activate \
    && pip3 install -r requirements/pt2.txt \
    && pip3 install . \
    && pip3 install -e git+https://github.com/Stability-AI/datapipelines.git@main#egg=sdata \
    && pip install hatch \
    && hatch build -t wheel   

[akx]

@metter You don't need to install all of sgm's dependencies – the webui doesn't install any of them and things do work. In all honesty, a dockerfile should just run python launch.py --exit – that would prepare everything, and then exit.

For one, that dockerfile clones taming-transformers, which is not necessary at all anymore (#10856).

[metter]

@metter You don't need to install all of sgm's dependencies – the webui doesn't install any of them and things do work. In all honesty, a dockerfile should just run python launch.py --exit – that would prepare everything, and then exit.

For one, that dockerfile clones taming-transformers, which is not necessary at all anymore (#10856).

Thank you for letting me know. I just tried it and I am getting this error:

=> ERROR [stage-1 10/35] RUN python launch.py --skip-torch-cuda-test  --exit                                                                                  4.9s
------                                                                                                                                                              
 > [stage-1 10/35] RUN python launch.py --skip-torch-cuda-test  --exit:                                                                                             
#25 0.535 Python 3.10.12 (main, Jun 13 2023, 15:51:40) [GCC 8.3.0]                                                                                                  
#25 0.535 Version: v1.5.1                                                                                                                                           
#25 0.535 Commit hash: 68f336bd994bed5442ad95bad6b6ad5564a5409a                                                                                                     
#25 0.537 Installing clip                                                                                                                                           
#25 4.025 Fetching updates for Stable Diffusion...
#25 4.877 Checking out commit for Stable Diffusion with hash: cf1d67a6fd5ea1aa600c4df58e5b47da45f6bdbf...
#25 4.887 fatal: reference is not a tree: cf1d67a6fd5ea1aa600c4df58e5b47da45f6bdbf
#25 4.888 Traceback (most recent call last):
#25 4.888   File "/stable-diffusion-webui/launch.py", line 39, in <module>
#25 4.888     main()
#25 4.888   File "/stable-diffusion-webui/launch.py", line 30, in main
#25 4.888     prepare_environment()
#25 4.888   File "/stable-diffusion-webui/modules/launch_utils.py", line 345, in prepare_environment
#25 4.888     git_clone(stable_diffusion_repo, repo_dir('stable-diffusion-stability-ai'), "Stable Diffusion", stable_diffusion_commit_hash)
#25 4.888   File "/stable-diffusion-webui/modules/launch_utils.py", line 156, in git_clone
#25 4.888     run(f'"{git}" -C "{dir}" checkout {commithash}', f"Checking out commit for {name} with hash: {commithash}...", f"Couldn't checkout commit {commithash} for {name}", live=True)
#25 4.888   File "/stable-diffusion-webui/modules/launch_utils.py", line 113, in run
#25 4.888     raise RuntimeError("\n".join(error_bits))
#25 4.888 RuntimeError: Couldn't checkout commit cf1d67a6fd5ea1aa600c4df58e5b47da45f6bdbf for Stable Diffusion.
#25 4.888 Command: "git" -C "/stable-diffusion-webui/repositories/stable-diffusion-stability-ai" checkout cf1d67a6fd5ea1aa600c4df58e5b47da45f6bdbf
#25 4.888 Error code: 128
------
executor failed running [/bin/bash -o pipefail -c python launch.py --skip-torch-cuda-test  --exit]: exit code: 1

Not quite getting why I get it because I dont have any issues installing A111 locally on my windows machine. cf1d67a6fd5ea1aa600c4df58e5b47da45f6bdbf is an actual commit though

[Najeebahmed11]

Hi @metter. can you please share complete docker file with me. As I am facing same issue in my docker build.

[metter]

Hi @metter. can you please share complete docker file with me. As I am facing same issue in my docker build.

sure. Here you go

# ---------------------------------------------------------------------------- #
#                         Stage 1: Download the models                         #
# ---------------------------------------------------------------------------- #
FROM alpine/git:2.36.2 as download

COPY builder/clone.sh /clone.sh

# Clone the repos and clean unnecessary files
#RUN . /clone.sh taming-transformers https://github.com/CompVis/taming-transformers.git 24268930bf1dce879235a7fddd0b2355b84d7ea6 && \
#    rm -rf data assets **/*.ipynb

RUN . /clone.sh stable-diffusion-stability-ai https://github.com/Stability-AI/stablediffusion.git cf1d67a6fd5ea1aa600c4df58e5b47da45f6bdbf && \
    rm -rf assets data/**/*.png data/**/*.jpg data/**/*.gif

RUN . /clone.sh CodeFormer https://github.com/sczhou/CodeFormer.git c5b4593074ba6214284d6acd5f1719b6c5d739af && \
    rm -rf assets inputs

RUN . /clone.sh BLIP https://github.com/salesforce/BLIP.git 48211a1594f1321b00f14c9f7a5b4813144b2fb9 && \
    . /clone.sh k-diffusion https://github.com/crowsonkb/k-diffusion.git 5b3af030dd83e0297272d861c19477735d0317ec && \
    . /clone.sh clip-interrogator https://github.com/pharmapsychotic/clip-interrogator 2486589f24165c8e3b303f84e9dbbea318df83e8 && \
    . /clone.sh generative-models https://github.com/Stability-AI/generative-models.git 45c443b316737a4ab6e40413d7794a7f5657c19f

WORKDIR /download
RUN wget -O model.safetensors https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors 
RUN wget -O sdxl_vae.safetensors https://huggingface.co/stabilityai/sdxl-vae/resolve/main/sdxl_vae.safetensors 



# ---------------------------------------------------------------------------- #
#                         Stage 2: Build the final image                       #
# ---------------------------------------------------------------------------- #
FROM python:3.10-buster

ENV DEBIAN_FRONTEND=noninteractive \
    PIP_PREFER_BINARY=1 \
    ROOT=/stable-diffusion-webui \
    PYTHONUNBUFFERED=1 

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

USER root
RUN apt-get update && \
    apt install -y \
    fonts-dejavu-core rsync git jq moreutils aria2 wget libgoogle-perftools-dev procps nano mc fish && \
    apt-get autoremove -y && rm -rf /var/lib/apt/lists/* && apt-get clean -y

ENV LD_PRELOAD=libtcmalloc.so  

# Clone the repository
RUN git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git /stable-diffusion-webui

WORKDIR /stable-diffusion-webui

# Reset to the specific commit and install requirements
RUN git reset --hard 68f336bd994bed5442ad95bad6b6ad5564a5409a && \
    pip install -r requirements_versions.txt

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

#copy from download stage
COPY --from=download /repositories/ ${ROOT}/repositories/
COPY --from=download /download/model.safetensors ${ROOT}/model.safetensors
COPY --from=download /download/sdxl_vae.safetensors ${ROOT}/models/Stablediffusion/VAE/sdxl_vae.safetensors

# Install generative models
#WORKDIR /stable-diffusion-webui/repositories/generative-models

# Install required packages from pypi inside the virtual environment
#RUN python3 -m venv .pt2
#RUN . .pt2/bin/activate \
#    && pip3 install -r requirements/pt2.txt \
#    && pip3 install . \
#    && pip3 install -e git+https://github.com/Stability-AI/datapipelines.git@main#egg=sdata \
#    && pip install hatch \
#    && hatch build -t wheel     

RUN python launch.py --skip-torch-cuda-test  --exit

# Create a directory for the interrogator data and copy the files
RUN mkdir ${ROOT}/interrogate && cp ${ROOT}/repositories/clip-interrogator/data/* ${ROOT}/interrogate

# Install CodeFormer dependencies
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r ${ROOT}/repositories/CodeFormer/requirements.txt  

# Set the working directory in the container
WORKDIR /stable-diffusion-webui/repositories/CodeFormer/weights/facelib/
# Download the file using wget
RUN wget https://github.com/sczhou/CodeFormer/releases/download/v0.1.0/parsing_parsenet.pth    

# Install generative models dependencies
WORKDIR ${ROOT}/repositories/generative-models
RUN --mount=type=cache,target=/root/.cache/pip \
    python3 -m venv .pt2 \
    && . .pt2/bin/activate \
    && pip3 install -r requirements/pt2.txt \
    && pip3 install . \
    && pip3 install -e git+https://github.com/Stability-AI/datapipelines.git@main#egg=sdata \
    && pip install hatch \
    && hatch build -t wheel    

WORKDIR /    

# Install Python dependencies (Worker Template)
COPY builder/requirements.txt /requirements.txt
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --upgrade pip && \
    pip install --upgrade -r /requirements.txt --no-cache-dir && \
    rm /requirements.txt

# Switch to the specified SHA
ARG SHA=68f336bd994bed5442ad95bad6b6ad5564a5409a 
RUN --mount=type=cache,target=/root/.cache/pip \
    cd stable-diffusion-webui && \
    git fetch && \
    git reset --hard ${SHA} && \
    pip install -r requirements_versions.txt

# Add the source files to the working directory
ADD src .
ADD test_inputs_folder /test_input

# Copy the cache.py script and run the cache step
WORKDIR /stable-diffusion-webui
RUN pip uninstall -y torchvision
RUN pip install  torchvision --index-url https://download.pytorch.org/whl/cu118
RUN pip uninstall -y torch
RUN pip install  torch --index-url https://download.pytorch.org/whl/cu118    
COPY builder/cache.py /stable-diffusion-webui/cache.py
RUN python cache.py --use-cpu=all --ckpt model.safetensors

WORKDIR /stable-diffusion-webui/extensions

# Clone some extensions
#RUN git clone https://github.com/Mikubill/sd-webui-controlnet.git
#RUN git clone https://github.com/Extraltodeus/multi-subject-render.git

# Install sd-webui-controlnet dependencies
#RUN --mount=type=cache,target=/root/.cache/pip \
#    pip install -r ${ROOT}/extensions/sd-webui-controlnet/requirements.txt

WORKDIR /

# Copy the models and embeddings directories from the host to the container
COPY models/Lora /stable-diffusion-webui/models/Lora
#COPY models/ControlNet /stable-diffusion-webui/models/ControlNet
#COPY models/openpose /stable-diffusion-webui/models/openpose
COPY embeddings /stable-diffusion-webui/embeddings

# Cleanup section (Worker Template)
RUN apt-get autoremove -y && \
    apt-get clean -y && \
    rm -rf /var/lib/apt/lists/*

RUN chmod +x /start.sh
CMD /start.sh