Controlnet training #2545

Ttl · 2023-03-03T14:09:53Z

Adds controlnet training code, script for adding controlnet to an existing model and short readme.

I trained successfully a circle filling controlnet with this code using GPU with 8 GB VRAM.

Works with circle dataset: https://github.com/lllyasviel/ControlNet/blob/main/docs/train.md

Control image should be in 0..1 range.

HuggingFaceDocBuilderDev · 2023-03-03T14:14:38Z

The documentation is not available anymore as the PR was closed or merged.

examples/controlnet/train_controlnet.py

examples/controlnet/add_controlnet.py

patrickvonplaten

Wow super cool job @Ttl !

Think we can merge this very soon - @williamberman would you like to take a look as well and test the script a bit? :-)

It would also be nice if:

We only save the controlnet to avoid that every training run copies all the SDv1-5 weights again. We should then maybe also automatically create a READEM that marks SDv1-5 as the base model. See here and here
It would be great to only have one training script that does everything in once - we usually don't add auxiliary scripts
Could we maybe also some image logging to the script and run the pipeline in the end? Could we show some examples of the model in the README?

Super excited about this!

Could we maybe add a bit more data to the

Ttl · 2023-03-03T16:28:34Z

I'm not sure how useful the automatic upload to hub is in this case. Training a controlnet requires much more work than Dreambooth or textual inversion. It requires making a new dataset and the amount of required training is much higher. I haven't used it myself and just copied it from existing training scripts. In my opinion it could be removed entirely.

isamu-isozaki · 2023-03-03T18:01:30Z

Thanks for this pr!

examples/controlnet/train_controlnet.py

williamberman · 2023-03-06T22:57:25Z

"During the training, we randomly replace 50% text prompts ct with empty strings"

Maybe I'm missing it, but I don't see text prompts being dropped?

williamberman · 2023-03-06T22:59:42Z

I see a few cli args that aren't used such as sample_batch_size and prior_loss_weight. Can we make sure that the config only applies to this training script?

examples/controlnet/train_controlnet.py

Ttl · 2023-03-07T16:35:38Z

Text prompts are not dropped. I didn't see it used in the original training code with this dataset.

I looked into image logging but I don't have enough VRAM to make it work. It should be possible to free enough VRAM to run the pipeline but it doesn't seem to work automatically.

Automatic readme generation for upload isn't maybe too useful for controlnet since they require preprocessor for the control image and it isn't possible to automatically document it.

Co-authored-by: Sayak Paul <[email protected]>

examples/controlnet/README.md

.gitignore

examples/controlnet/.gitignore

examples/controlnet/README.md

src/diffusers/models/controlnet.py

examples/controlnet/train_controlnet.py

patrickvonplaten · 2023-03-14T21:32:06Z

Amazing job @Ttl and @williamberman - thanks a lot for putting this together - this will be very useful!

@williamberman could you maybe upload the images to a dataset on the Hub and load from there? We don't want to make the github heavy by adding images (this would otherwise be in the git history forever)

Apart from this good to merge for me!

Co-authored-by: Patrick von Platen <[email protected]>

giorgio130 · 2023-04-04T12:23:02Z

Thanks a lot for this example! I noticed that when using xformers==0.0.17, the training for the circle filling example does not seem to get anywhere, according to https://github.com/lllyasviel/ControlNet/blob/main/docs/train.md with a batch size of 4 it should already give sensible results after 4000 steps, but in my case I'm at 8000 and still it fumbles with circles of random sizes and positions. Has anyone tried this combination successfully?

patrickvonplaten · 2023-04-04T15:27:04Z

cc @yiyixuxu here

* Controlnet training code initial commit Works with circle dataset: https://github.com/lllyasviel/ControlNet/blob/main/docs/train.md * Script for adding a controlnet to existing model * Fix control image transform Control image should be in 0..1 range. * Add license header and remove more unused configs * controlnet training readme * Allow nonlocal model in add_controlnet.py * Formatting * Remove unused code * Code quality * Initialize controlnet in training script * Formatting * Address review comments * doc style * explicit constructor args and submodule names * hub dataset NOTE - not tested * empty prompts * add conditioning image * rename * remove instance data dir * image_transforms -> -1,1 . conditioning_image_transformers -> 0, 1 * nits * remove local rank config I think this isn't necessary in any of our training scripts * validation images * proportion_empty_prompts typo * weight copying to controlnet bug * call log validation fix * fix * gitignore wandb * fix progress bar and resume from checkpoint iteration * initial step fix * log multiple images * fix * fixes * tracker project name configurable * misc * add controlnet requirements.txt * update docs * image labels * small fixes * log validation using existing models for pipeline * fix for deepspeed saving * memory usage docs * Update examples/controlnet/train_controlnet.py Co-authored-by: Sayak Paul <[email protected]> * Update examples/controlnet/train_controlnet.py Co-authored-by: Sayak Paul <[email protected]> * Update examples/controlnet/README.md Co-authored-by: Sayak Paul <[email protected]> * Update examples/controlnet/README.md Co-authored-by: Sayak Paul <[email protected]> * Update examples/controlnet/README.md Co-authored-by: Sayak Paul <[email protected]> * Update examples/controlnet/README.md Co-authored-by: Sayak Paul <[email protected]> * Update examples/controlnet/README.md Co-authored-by: Sayak Paul <[email protected]> * Update examples/controlnet/README.md Co-authored-by: Sayak Paul <[email protected]> * Update examples/controlnet/README.md Co-authored-by: Sayak Paul <[email protected]> * Update examples/controlnet/README.md Co-authored-by: Sayak Paul <[email protected]> * remove extra is main process check * link to dataset in intro paragraph * remove unnecessary paragraph * note on deepspeed * Update examples/controlnet/README.md Co-authored-by: Patrick von Platen <[email protected]> * assert -> value error * weights and biases note * move images out of git * remove .gitignore --------- Co-authored-by: William Berman <[email protected]> Co-authored-by: Sayak Paul <[email protected]> Co-authored-by: Patrick von Platen <[email protected]>

Ttl added 8 commits March 2, 2023 19:21

Controlnet training code initial commit

aac7961

Works with circle dataset: https://github.com/lllyasviel/ControlNet/blob/main/docs/train.md

Script for adding a controlnet to existing model

6985732

Fix control image transform

da73a3e

Control image should be in 0..1 range.

Add license header and remove more unused configs

6501ca2

controlnet training readme

62fdaf8

Allow nonlocal model in add_controlnet.py

7a92d6f

Formatting

fcbe52e

Remove unused code

87e67bc

Code quality

391112e

patrickvonplaten reviewed Mar 3, 2023

View reviewed changes

examples/controlnet/train_controlnet.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Mar 3, 2023

View reviewed changes

examples/controlnet/train_controlnet.py Show resolved Hide resolved

patrickvonplaten reviewed Mar 3, 2023

View reviewed changes

examples/controlnet/train_controlnet.py Show resolved Hide resolved

patrickvonplaten reviewed Mar 3, 2023

View reviewed changes

examples/controlnet/train_controlnet.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Mar 3, 2023

View reviewed changes

examples/controlnet/add_controlnet.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Mar 3, 2023

View reviewed changes

patrickvonplaten requested a review from williamberman March 3, 2023 15:16

Ttl added 2 commits March 3, 2023 18:19

Initialize controlnet in training script

6f34077

Formatting

c375d7b

williamberman reviewed Mar 6, 2023

View reviewed changes