[Bug]: DMD2_SDXL_4step LoRA significantly increases generation time with SDXL model

[biandh]

Checklist

• The issue exists after disabling all extensions
• The issue exists on a clean installation of webui
• The issue is caused by an extension, but I believe it is caused by a bug in the webui
• The issue exists in the current version of the webui
• The issue has not been reported before recently
• The issue has been reported before but has not been fixed yet

What happened?

Issue Description

When using DMD2_SDXL_4step LoRA with SDXL 1.0 base model, the image generation time increases significantly from ~10 seconds to several minutes.

Environment

• WebUI: 82a973c (2024-07-27)
• Base Model: stable-diffusion-xl-base-1.0 (https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/tree/main sd_xl_base_1.0.safetensors)
• LoRA: DMD2_SDXL_4step (https://huggingface.co/tianweiy/DMD2 Lora file: dmd2_sdxl_4step_lora.safetensors)
• GPU: GeForce RTX 4090
• Launch Arguments: --theme dark --xformers --no-half-vae --api --autolaunch --skip-python-version-check

Steps to Reproduce

1. Load SDXL 1.0 base model
2. Generate an image without LoRA (takes ~10 seconds)
3. Add DMD2_SDXL_4step LoRA
4. Generate the same image (now takes several minutes)

Expected Behavior

DMD2_SDXL_4step LoRA should accelerate the generation process.

Actual Behavior

Generation time increases significantly when using the LoRA.

Steps to reproduce the problem

Steps to Reproduce

1. Load SDXL 1.0 base model
2. Generate an image without LoRA (takes ~10 seconds)
3. Add DMD2_SDXL_4step LoRA
4. Generate the same image (now takes several minutes)

What should have happened?

Expected Behavior

DMD2_SDXL_4step LoRA should accelerate the generation process.

What browsers do you use to access the UI ?

No response

Sysinfo

Console logs

SD-WebUI Launcher Diagnostic File

Date: 2024-11-08 00:56:26
Launcher Version: 2.8.10.387
Data File Version: 2024-10-27 12:33
SD-WebUI Version: 82a973c04367123ae98bd9abdf80d9eda9b910e2 (2024-07-27 20:49:39)
Working Directory: C:\Users\vipuser\Desktop\sd-webui-aki\sd-webui-aki-v4.9.1
------------------------
System Information: 
OS: Microsoft Windows NT 10.0.19044.0
CPU: 12 cores
Memory Size: 29839 MB
Page File Size: 12104 MB

NVIDIA Management Library:
  NVIDIA Driver Version: 528.49
  NVIDIA Management Library Version: 12.528.49

CUDA Driver:
  Version: 12000
  Devices: 
    00000000:00:08.0 0: NVIDIA GeForce RTX 4090 [89] 23 GB

NvApi:
  Version: 52849 r528_37

HIP Driver:
  Not Available

DirectML Driver: 
  Devices: 
    9860 0: NVIDIA GeForce RTX 4090 23 GB
    9860 1: NVIDIA GeForce RTX 4090 23 GB
    140 2: Microsoft Basic Render Driver 0 GB
    9860 3: NVIDIA GeForce RTX 4090 23 GB

Intel Level Zero Driver:
  Not Available

------------------------
Environment Variables: 
FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer
TEMP=C:\Users\vipuser\AppData\Local\Temp
USERPROFILE=C:\Users\vipuser
TMP=C:\Users\vipuser\AppData\Local\Temp
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
SESSIONNAME=RDP-Tcp#2
PUBLIC=C:\Users\Public
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_LEVEL=6
CLIENTNAME=EASTBLUEE499
NUMBER_OF_PROCESSORS=12
LOCALAPPDATA=C:\Users\vipuser\AppData\Local
DriverData=C:\Windows\System32\Drivers\DriverData
SystemDrive=C:
USERDOMAIN_ROAMINGPROFILE=galaxy-GUZaFCY
ALLUSERSPROFILE=C:\ProgramData
ComSpec=C:\Windows\system32\cmd.exe
CommonProgramFiles=C:\Program Files\Common Files
FPS_BROWSER_USER_PROFILE_STRING=Default
ProgramFiles=C:\Program Files
Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Git\cmd;C:\ProgramData\Anaconda3;C:\ProgramData\Anaconda3\Lib\site-packages\win32;C:\ProgramData\Anaconda3\Lib\site-packages\pywin32_system32;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\dotnet\;C:\Users\vipuser\Desktop\ffmpeg-2024-10-10-git-0f5592cfc7-full_build\bin;C:\Users\vipuser\AppData\Local\Programs\Python\Python38\Scripts\;C:\Users\vipuser\AppData\Local\Programs\Python\Python38\;C:\Users\vipuser\AppData\Local\Microsoft\WindowsApps;;C:\Users\vipuser\AppData\Local\Programs\Microsoft VS Code\bin
OS=Windows_NT
APPDATA=C:\Users\vipuser\AppData\Roaming
ProgramFiles(x86)=C:\Program Files (x86)
HOMEDRIVE=C:
PROCESSOR_REVISION=8f08
PSModulePath=C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
SystemRoot=C:\Windows
HOMEPATH=\Users\vipuser
COMPUTERNAME=galaxy-GUZaFCY
ProgramData=C:\ProgramData
windir=C:\Windows
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 143 Stepping 8, GenuineIntel
CommonProgramW6432=C:\Program Files\Common Files
ProgramW6432=C:\Program Files
PROCESSOR_ARCHITECTURE=AMD64
USERNAME=vipuser
USERDOMAIN=galaxy-GUZaFCY
LOGONSERVER=\\galaxy-GUZaFCY
------------------------
Log: 
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: v1.10.1
Commit hash: 82a973c04367123ae98bd9abdf80d9eda9b910e2

Launching Web UI with arguments: --theme dark --xformers --no-half-vae --api --autolaunch --skip-python-version-check
Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will be limited to those added through the extra networks menu.
[-] ADetailer initialized. version: 24.9
.
0, num models: 10

ControlNet preprocessor location: C:\Users\vipuser\Desktop\sd-webui-aki\sd-webui-aki-v4.9.1\extensions\sd-webui-controlnet\annotator\downloads
2024-11-08 00:26:40,184 - ControlNet - �[0;32mINFO�[0m - ControlNet v1.1.455
sd-webui-prompt-all-in-one background API service started successfully.
Loading weights [31e35c80fc] from C:\Users\vipuser\Desktop\sd-webui-aki\sd-webui-aki-v4.9.1\models\Stable-diffusion\sdxl1.0\sd_xl_base_1.0.safetensors
[LyCORIS]-�[0;33mWARNING�[0m: LyCORIS legacy extension is now loaded, if you don't expext to see this message, please disable this extension.
Creating model from config: C:\Users\vipuser\Desktop\sd-webui-aki\sd-webui-aki-v4.9.1\repositories\generative-models\configs\inference\sd_xl_base.yaml

Applying attention optimization: xformers... done.
WARNING:py.warnings:C:\Users\vipuser\Desktop\sd-webui-aki\sd-webui-aki-v4.9.1\python\lib\site-packages\torch\nn\functional.py:5504: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.)
  attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)

Model loaded in 12.1s (load weights from disk: 0.4s, load config: 0.4s, create model: 2.0s, apply weights to model: 8.3s, apply half(): 0.1s, move model to device: 0.3s, calculate empty prompt: 0.4s).
2024-11-08 00:26:55,575 - ControlNet - �[0;32mINFO�[0m - ControlNet UI callback registered.
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

�[92mIIB Database file has been successfully backed up to the backup folder.�[0m
Startup time: 64.6s (prepare environment: 18.5s, import torch: 10.5s, import gradio: 3.2s, setup paths: 1.6s, initialize shared: 0.6s, other imports: 1.7s, list SD models: 0.1s, load scripts: 8.0s, create ui: 14.5s, gradio launch: 3.3s, add APIs: 2.0s, app_started_callback: 0.4s).
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 1159653272, "all_seeds": [1159653272], "subseed": 3150971306, "all_subseeds": [3150971306], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "DPM++ 2M", "cfg_scale": 7, "steps": 20, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {"Schedule type": "Karras"}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 1159653272, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108002938", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,", "all_prompts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 1372934445, "all_seeds": [1372934445], "subseed": 2811219341, "all_subseeds": [2811219341], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 7, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {"Schedule type": "Karras"}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,\nSteps: 6, Sampler: LCM, Schedule type: Karras, CFG scale: 7, Seed: 1372934445, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003312", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,", "all_prompts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 464245114, "all_seeds": [464245114], "subseed": 1711710803, "all_subseeds": [1711710803], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 7, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,\nSteps: 6, Sampler: LCM, Schedule type: Automatic, CFG scale: 7, Seed: 464245114, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003419", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,", "all_prompts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 1608476397, "all_seeds": [1608476397], "subseed": 3075197645, "all_subseeds": [3075197645], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 1, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,\nSteps: 6, Sampler: LCM, Schedule type: Automatic, CFG scale: 1, Seed: 1608476397, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003508", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,", "all_prompts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 3853543390, "all_seeds": [3853543390], "subseed": 813517798, "all_subseeds": [813517798], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 1, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,<lora:dmd2_sdxl_4step_lora:1>,\nSteps: 6, Sampler: LCM, Schedule type: Automatic, CFG scale: 1, Seed: 3853543390, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003556", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 2808798004, "all_seeds": [2808798004], "subseed": 1746927857, "all_subseeds": [1746927857], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 1, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 6, Sampler: LCM, Schedule type: Automatic, CFG scale: 1, Seed: 2808798004, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003709", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 3580553760, "all_seeds": [3580553760], "subseed": 3937475132, "all_subseeds": [3937475132], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 1, "steps": 6, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 6, Sampler: LCM, Schedule type: Automatic, CFG scale: 1, Seed: 3580553760, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003723", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 3002655950, "all_seeds": [3002655950], "subseed": 4033832068, "all_subseeds": [4033832068], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "LCM", "cfg_scale": 1, "steps": 20, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 20, Sampler: LCM, Schedule type: Automatic, CFG scale: 1, Seed: 3002655950, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003734", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 2250936571, "all_seeds": [2250936571], "subseed": 3628021471, "all_subseeds": [3628021471], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "DPM++ 2M", "cfg_scale": 1, "steps": 20, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {"Schedule type": "Karras"}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 1, Seed: 2250936571, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003755", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 2441762049, "all_seeds": [2441762049], "subseed": 1188052010, "all_subseeds": [1188052010], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "DPM++ 2M", "cfg_scale": 7, "steps": 20, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {"Schedule type": "Karras"}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 2441762049, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003814", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}
{"prompt": "the doctor is treating the patient,", "all_prompts": ["the doctor is treating the patient,"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 2947229270, "all_seeds": [2947229270], "subseed": 135279198, "all_subseeds": [135279198], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "DPM++ 2M", "cfg_scale": 7, "steps": 20, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_name": "sd_xl_base_1.0", "sd_model_hash": "31e35c80fc", "sd_vae_name": null, "sd_vae_hash": null, "seed_resize_from_w": -1, "seed_resize_from_h": -1, "denoising_strength": 0.7, "extra_generation_params": {"Schedule type": "Karras"}, "index_of_first_image": 0, "infotexts": ["the doctor is treating the patient,\nSteps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 2947229270, Size: 512x512, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, Version: v1.10.1"], "styles": [], "job_timestamp": "20241108003824", "clip_skip": 2, "is_using_inpainting_conditioning": false, "version": "v1.10.1"}

Additional information

No response