Saving error finetuning stable diffusion LoRA

[sr5434]

Describe the bug

After the stable diffusion model is fully trained, an error occurs:

Traceback (most recent call last):
  File "train_text_to_image_lora.py", line 872, in <module>
    main()
  File "train_text_to_image_lora.py", line 825, in main
    unet.save_attn_procs(args.output_dir)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/loaders.py", line 273, in save_attn_procs
    weights_no_suffix = weights_name.replace(".bin", "")
AttributeError: 'NoneType' object has no attribute 'replace'

After this error occurs, the weights for the model aren't saved.

Reproduction

I ran the below command in Google Colab:

accelerate launch train_text_to_image_lora.py --mixed_precision="fp16" --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" --dataset_name="Norod78/microsoft-fluentui-emoji-512-whitebg" --caption_column="text" --resolution=512 --train_batch_size=2 --num_train_epochs=1 --output_dir="./sd-model-finetuned-lora" --max_train_steps=1000 --checkpointing_steps=500 --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 --seed=42

Logs

Traceback (most recent call last):
  File "train_text_to_image_lora.py", line 872, in <module>
    main()
  File "train_text_to_image_lora.py", line 825, in main
    unet.save_attn_procs(args.output_dir)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/loaders.py", line 273, in save_attn_procs
    weights_no_suffix = weights_name.replace(".bin", "")
AttributeError: 'NoneType' object has no attribute 'replace'

System Info

Google colab running the latest version of diffusers.

Is this a bug, or is there something I have to do on my end to fix this?

[StarrickLiu]

I have a same issue...
How to fix it, plz?

[wangjuns]

Me too.
It's seem that move below code upper then clean up code in diffusers/loaders.py will fix bug.

 if weights_name is None:
            if safe_serialization:
                weights_name = LORA_WEIGHT_NAME_SAFE
            else:
                weights_name = LORA_WEIGHT_NAME

[wangjuns]

I have a same issue... How to fix it, plz?

you could bypass this bug by modify train_text_to_image_lora.py.

diff --git a/examples/dreambooth/train_dreambooth_lora.py b/examples/dreambooth/train_dreambooth_lora.py
index c9321982..db26a1bf 100644
--- a/examples/dreambooth/train_dreambooth_lora.py
+++ b/examples/dreambooth/train_dreambooth_lora.py
@@ -987,7 +987,7 @@ def main(args):
     accelerator.wait_for_everyone()
     if accelerator.is_main_process:
         unet = unet.to(torch.float32)
-        unet.save_attn_procs(args.output_dir)
+        unet.save_attn_procs(args.output_dir, weights_name='xyz.bin')

         # Final inference
         # Load previous pipeline
@@ -998,7 +998,7 @@ def main(args):
         pipeline = pipeline.to(accelerator.device)

         # load attention processors
-        pipeline.unet.load_attn_procs(args.output_dir)
+        pipeline.unet.load_attn_procs(args.output_dir, weights_name='xyz.bin')

         # run inference
         if args.validation_prompt and args.num_validation_images > 0:

[sr5434]

Thank you, that seems to have fixed the problem.

[douo]

pipeline.unet.load_attn_procs(args.output_dir, weights_name='xyz.bin')

It seem to be weight_name(ref) not weights_name .

[sr5434]

@wangjuns you should consider opening a PR with those changes.

[sr5434]

The error also occurs when loading weights:

AttributeError                            Traceback (most recent call last)
[<ipython-input-2-417d63d04d54>](https://localhost:8080/#) in <module>
      4 model_path = "/content/sd-model-finetuned-lora/pytorch_lora_weights.bin"
      5 pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
----> 6 pipe.unet.load_attn_procs(model_path)
      7 pipe.to("cuda")
      8 

[/usr/local/lib/python3.9/dist-packages/diffusers/loaders.py](https://localhost:8080/#) in load_attn_procs(self, pretrained_model_name_or_path_or_dict, **kwargs)
    151         model_file = None
    152         if not isinstance(pretrained_model_name_or_path_or_dict, dict):
--> 153             if (is_safetensors_available() and weight_name is None) or weight_name.endswith(".safetensors"):
    154                 if weight_name is None:
    155                     weight_name = LORA_WEIGHT_NAME_SAFE

AttributeError: 'NoneType' object has no attribute 'endswith'

[patrickvonplaten]

Looks like we need to fix something here! @sr5434 could you maybe add a link to the trained LoRA checkpoints ? Also cc @sayakpaul could you maybe try to have a look into this? :-)

[sayakpaul]

This is related to #2616.

@wfng92 proposed a nice suggestion here: #2616 (comment).

The easiest workaround for now seems to be #2616 (comment).

[patrickvonplaten]

Does this fix the problem: https://github.com/huggingface/diffusers/pull/2655/files ?

[sr5434]

@sayakpaul I tried the work around, but now the error occurs at load time. @patrickvonplaten I will send the LoRA weights as soon as I can.

[0xdigiscore]

@patrickvonplaten

!pip install git+https://github.com/huggingface/diffusers@correct_lora_saving_loading
!accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \
  --pretrained_model_name_or_path="Linaqruf/anything-v3.0" \
  --dataset_name="ethers/azuki-datasets" --caption_column="text" \
  --resolution=512 --random_flip \
  --train_batch_size=1 \
  --max_train_steps=1 \
  --checkpointing_steps=1000 \
  --validation_epochs=100 \
  --num_validation_images=0 \
  --output_dir="/content/drive/MyDrive/differ-out" \
  --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --seed=42 \
  --max_train_samples=10

An error is still reported when the training result is saved in colab.

[0xdigiscore]

@patrickvonplaten here is my colab: https://colab.research.google.com/drive/1Z-Xt8c4Wjk5d3ft5VoC0P-J0y0427i4z?usp=sharing
you can use it for testing

[sayakpaul]

@0xhelloweb3 might have been an improper dependency resolution problem in your Colab.

Mine worked fine: https://colab.research.google.com/gist/sayakpaul/a6640268902e3e75aef58b5c9f06f042/scratchpad.ipynb

[0xdigiscore]

@0xhelloweb3 might have been an improper dependency resolution problem in your Colab.

Mine worked fine: https://colab.research.google.com/gist/sayakpaul/a6640268902e3e75aef58b5c9f06f042/scratchpad.ipynb

Thx, he problem has been solved and it works normally

[sayakpaul]

@sr5434 could you check if #2655 solves your issue and close this issue accordingly?

[sr5434]

Yes, #2655 fixes the issue. Thank you @patrickvonplaten for creating the PR.