[Feature Request] Control model offsets and some others

[lllyasviel]

Hi great plugin. I played with it a bit and the memory optimization is really good. Some considerations should be put with higher priority:

1. The requirements for some mode like the segmentation is broken.

2. We really need a button to offset the weight inside the controlnet: that will immediately solve the distorted faces in pose mode and distorted edges in canny mode. This can be done by offsetting the weight copy inside the controlnet with user models and sd15. Right now all controls are targeted to SD15. These should be retargeted to user models.

[lllyasviel]

or at least we should put some control_any3_X.pth files for anime models. I cannot release these officially because of many considerations. But this can be done by 3rd projects.
But a button to directly compute it from user model is better.

[lllyasviel]

the formulation is

Any3.control_model.weights 
= SD15.control_model.weights + Any3.model.diffusion_model.weights - SD15.model.diffusion_model.weights

[lllyasviel]

or we can directly store

SD15.control_model.weights - SD15.model.diffusion_model.weights

and any time user load the model, it add the base weight from user model

[lllyasviel]

Updated lllyasviel/ControlNet#12

[lllyasviel]

This should be considered early since it seems that this plugin is super popular in both english and asian community. so the early this is fixed, the less things we will need to ask people to try download new files.

[Mikubill]

Thanks for pointing out. Will implement some fix immediately

[Mikubill]

Fixed in b9efb60, should work but still need some tests.

[lllyasviel]

seems broken, not working anymore

[catboxanon]

Seems to be working for me with this most recent change. I got Offset cloned: 298 values in my console.

[lllyasviel]

Great. Now working. But how is the current offset computed? I do not even have sd15 in my webui.

[catboxanon]

This is what it looks like when unchecking Apply transfer control when loading models in settings by the way, just to confirm there is a difference. Same seed.

[lllyasviel]

non-transfer results looks even better? perhaps because any3 is bad at drawing house

[lllyasviel]

by the way, what is the best practice to develop webui extension? open private github repo? directly write code in webui folder?

[Mikubill]

Hmm, looks like transferring control brings some instability. Temporary disabled and could be re-enable in the settings.

[brunogcar]

by the way, what is the best practice to develop webui extension? open private github repo? directly write code in webui folder?

https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Developing-extensions

https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Extensions

[kohya-ss]

Hi, thank you for great work!

or we can directly store

SD15.control_model.weights - SD15.model.diffusion_model.weights

and any time user load the model, it add the base weight from user model

I implemented control transfer with this approach. The difference is calculated in advance and stored to the file. The implementation is here:

https://github.com/kohya-ss/sd-webui-controlnet-lora/tree/support-lora

extract_controlnet_diff.py makes the difference and save the state_dic with key differece as a marker, and cldm.py handles it on the fly.

I can make a pull request, and I also think you can easily copy the codes from there. Please feel free to modify it.

(Please forget the name of the repo. I intended to support ControlNet with LoRA...)

[lllyasviel]

I am working on these. But I have also found that in some cases, non-transfered models even have better performance. Very weird. I'm trying to understand. We still know too little about neural networks.

[Mikubill]

I can make a pull request, and I also think you can easily copy the codes from there. Please feel free to modify it.

Feel free to make a PR.

[kohya-ss]

I've made the PR #80 :)

[Mikubill]

Merged. This method seems to be more correctly and effective.
btw, any samples?

[kohya-ss]

Thank you for merging!

As lllyasviel mentioned above, non-transferred models seem to have better or almost same performance sometimes. In my test, openpose seems to generate almost same image.

But canny and scribble are slightly better with transfer (little crisp image in canny, improved background in scribble). I'm using ACertainty for test.

Please let me know if you need image files or more samples.

Canny:

Without transfer:

With transfer:

Scribble:

Without transfer:

With transfer:

[lllyasviel]

perhaps those anime models are trained too much in anime domain and forgets many general object context concepts, and controlnet without transferring accidentally brought those general concepts back.

[kohya-ss]

I've uploaded pre-made difference files.
https://huggingface.co/kohya-ss/ControlNet-diff-modules/tree/main

I have checked each file by generating an image with the extension, but would be happy to check just to be sure.

[ljleb]

Should the code be tweaked to accept difference models along with (or instead of) full control checkpoints? Would save some disk space for some people, lighter to move around/download etc.

I'm just not sure how to distinguish diff models vs complete checkpoints programatically.

[Mikubill]

Difference models should be loaded without any issues. Feel free to open new issue if not works

[CCRcmcpe]

IMO, doing a weights merge will not save the bad generation quality, possibly will perform even worse. This is like saying human + w * (human - ape) = more intelligent human. Train a proper model based on any3 (etc.) is the way to go.

[Sansui233]

Thanks for your great work!
I tested my sketch on pre-made extracted canny models with Anything-v3-fp16. Weird that the non-diff models always works much better on details and more like "Anything's style". I don't understand much about that, just post the feedback.

Original sketch:

ControlNet:

[paulo-coronado]

Hello @Mikubill, @lllyasviel or @kohya-ss. Could you pls clarify me one thing? In this project are you replacing ControlNet base model with the user model (e.g. AnythingV3), or are you doing some other operation? I don't understand how Transfer Control happens "on the fly" without the need to generate a new merged model (e.g. AnythingV3ControlCanny.pth).

Could you please explain how "cldm.py handles it on the fly"? How is the user model merged with the SDControl model? Thanks!

[haofanwang]

@plcpinho Check my reply here. on the fly never means it doesn't merging, it merges in time once the model updated without saving the merged model locally.

[tavihalperin]

@haofanwang
Thanks for the clarifications on this!
So the the A1111 "on the fly" should match the output of the "statically merged" model right?
why do the results look better in this approach?

[catboxanon]

IMO, doing a weights merge will not save the bad generation quality, possibly will perform even worse. This is like saying human + w * (human - ape) = more intelligent human. Train a proper model based on any3 (etc.) is the way to go.

@CCRcmcpe I think I understand what you're saying, but it's been shown in other cases doing this type of merge as described in #73 (comment) drastically improves results. Take a look at this: https://old.reddit.com/r/StableDiffusion/comments/zyi24j/how_to_turn_any_model_into_an_inpainting_model/