SD Chad - Stable Diffusion Aesthetic Scorer

[grexzen]

SD Chad - Stable Diffusion Aesthetic Scorer

Have been using SD to create art for the last month, finding a template that works across prompt, seed, settings, and then creating 100s of images from it, selecting the best, deleting the rest.

That flow works great already, have lots of pics that look as good as those trending on ArtStation. Then I thought about automating this using AI. Here is what I have done so far:

• Got the 10M prompts, seeds, settings, urls from the SD beta discord bot, Aug 2022 (same set used for krea.ai and lexica.art)
  https://github.com/krea-ai/open-prompts
• Normalized the data by selecting 548K samples that are 512 x 512, 50 steps, 7 scale, not repeated seeds + prompt combo
  https://drive.google.com/file/d/1c4WHxtlzvHYd0UY5WCMJNn2EO-Aiv2A0/view
• Downloaded the images, got image and text embeddings using img2dataset and Clip Retrieval (amazing framework)
  https://github.com/rom1504/clip-retrieval
• Ran those images through ASV1 (created by LAION from 4K gens) and ASV2 (created from 250K gens + no gens, used to get the image set that SD was trained on)
  https://github.com/LAION-AI/aesthetic-predictor
  https://github.com/christophschuhmann/improved-aesthetic-predictor
• Created a script for SD, so one can generate images, and depending on score, they get saved in different folders, different file names
  https://github.com/grexzen/SD-Chad/blob/main/chad_scorer.py
• NEW Retrained the scoring model using 200K gens scored by people 1-10 to create a second comparing option (ASV2 = SD vs real images, CSV1 = SD vs SD)
  https://github.com/grexzen/SD-Chad/blob/main/chadscorer.pth
• NEW Researched the correlation between scores at step 1 and step 20 across 200K gens, not repeated seeds + prompt combos (small 37% positive correlation, not usable)

Now I am retraining the scoring model again using the top 2,500 images scored from 200K gens (1.5 model + new VAE + style = best gens so far), and handpicking the few images that I would personally published in my art channels. First test was great, the model really seems to understand which images I would pick.

SD Chad Script

• Add the script to SD by following these instructions
  https://github.com/grexzen/SD-Chad/blob/main/chad_scorer.py
  https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Custom-Scripts
• Select the script from the dropdown, and add how many gens you want to produce (this adds all gens above 6.9 to your outputs folder, that's the statistically significant score limit from 500K scored gens; if you use the chadscorer model instead, change the limit to 7.7)
  https://github.com/grexzen/SD-Chad/blob/main/chadscorer.pth
• Create gens from lists of prompts and seeds by using this script in the custom script option on SD (create 1 txt for prompt + 1 for seeds, add paths to script)
  https://github.com/grexzen/SD-Chad/blob/main/create_gens_from_list_of_prompts_and_seeds_and_score_them.py
• Retrain the scoring model by using the training scripts in this repo (create cvs with a list of images with 2 columns IMAGEPATH & AVERAGE_RATING, add paths to your images and scores, transform it into a parquet file, then run them).
  https://github.com/grexzen/SD-Chad/blob/main/simple_inference.py
  https://github.com/grexzen/SD-Chad/blob/main/train_predictor.py

ASV1 vs ASV2

Here is ASV1. Album score =10 https://ibb.co/album/cY7GQW. Album score = 0 https://ibb.co/album/84p0Bk.

Here is ASV2. Album score = 8 (highest) https://ibb.co/album/ypWyhL. Album score = 2 (lowest) https://ibb.co/album/0Rk3Yx.

ASV1 has a nicer distribution of scores, while ASV2 is pretty tight in the middle. Since ASV2 was created by scoring non-gens that might be why is so strict scoring gens from SD. It also seems to prefer realistic images.

ASV2 vs CSV1

ASV2 is great for scoring your images vs other gens + no gens (SD vs real pics) and CSV1 is great for scoring your images vs other gens only (SD vs SD).

Below are the distributions, average scores, and 2 & 3 standard deviations from the mean.

.

[Tollanador]

Given that the highest rated images still have poorly formed hands, I feel the the most useful part of this approach is to use the low scores as a filter. So ditch anything under a score of 4. Then maybe bucket the remaining into low/high potential buckets or similar.

Especially if the image generation can be done at step 5,15 and final (or similar).

I'd like to be able to use this with other scripts though. If you make it a script it means not being able to use the other scripts who's output may be what is wanting to have the aesthetic score generated.

Can you consider making this a non-script PR?

[grexzen]

That is how I have been using it at first, just to filter the noise, if you check out the current script it cuts anything below the 2 standard deviations from the mean to make it easier to deal with bulk generations.

https://github.com/grexzen/SD-Chad/blob/main/chad_scorer.py

I did a run of 10,000 generations using some of my best prompts and there were too many good images in the set, so this way, out of 10K, it kept the top 500.

Then it gets harder from there, since those are all awesome. I retrained it with only gens using 100K images scored by people, and seems to have a nicer distribution than ASV2 (have to post the new model yet).

Next I am going to train it to look at the top best and choose only a couple I would personally post in my art pages. For that I need to feed it at least 1000 images in the 'publish' vs 'not' classes.

I am not sure if I can do the non-script part, as I am literally not a dev, but autistic enough to drive things until they work : )

[Tollanador]

hi 5 for autism! (Except the shitty parts ha)

[Tollanador]

Hey, I played with your repo and found it can be sped up significantly by using CLIP-ONNX, https://github.com/Lednik7/CLIP-ONNX.
It's pretty straightforward to switch out the open_clip model with the Onnx version.
It made using the aesthetic scoring in CPU mode bearable.

[grexzen]

Nice! I am going to try it for sure, that is something that I thought it needed from the start.

The one thing to keep in mind, is that different models change the scores. For instance, I tried openclip H/14, the newest, best one, and it change the scores, so for benchmarking is tricky.

Dimensions change scores, models, xtransformers, embeddings, so many variables.

[fernicar]

the new feature "Skip" could benefit from this score prediction as soon as it get the low score, lets say a checkbox "Auto-Skip using SD Aesthetic Scorer"!

[grexzen]

Great idea, I'll play with the code on that part to see if I can hook it up. I already retrained it and I have a better scoring model (need to publish in the repo), and I am testing how fast in the steps you can correlate that to a good vs bad image down the final step.

I tested it on 10,000 gens at 5 steps, and it can score amazing pics by the hundreds w that few steps. But I want to see if it can do it in 1 : )

[fernicar]

maybe a setting group of some possible features:

• Checkbox: Auto-Skip using SD Aesthetic Scorer.
• Filepath: Nested folder for High Scored, or Low Scored.
• Slider: How much progress percentage before Scoring.
• Slider: How much progress percentage before Scoring second pass (highres-fix from txt2img).

[scrambled2]

Pardon my ignorance. Should I just git clone this repo into my main stable-diffusion-webui directory and drop your chad_scorer.py file under the script for the main repo?

[grexzen]

Yep, you can just pull autos repo, and then add the script to that folder.

I added more context above, and some new things, like a new model, how to train your own scorer, and how to create many gens from lists of prompts and seeds.

Creating your own scorer is best to be honest as a second filter (first anything below the main score is trash), because then you end up with tons of good gens that you still need to sort out.

With your personalized scorer, the scorer knows what YOU would pick personally. Works great. I have not made that into an easy to use script for autos repo, but you can follow the instructions above, it is not hard.

[scrambled2]

Awesome, thanks for the response, I greatly appreciate it. What is a solid amount of samples to have to train a scorer based on my own taste before I won't see much variation in the data?

[grexzen]

I used about 1,000 gens I chose manually and score them as a 9, then used 25,000 gens I didn't like or scored below the first filter limit of 7, and scored as a 3.

That gives a great distribution. Sorry I replied so late, but I don't check these notifications.

[pikachungus]

I don't understand how to get the windows tags for sorting like it was shown in the extension screenshot. Can somebody help?

[grexzen]

I think the dev that created the extension from this, wrote that the tags only work in Windows, and in his repo it seems that he wrote that a new update to the main repo broke the tags for now.

[iamianM]

Is there a chance this might turn into an extension to easily be used with the main UI? I'm very interested in using this.

[grexzen]

Someone else already used the base code to create an extension, check out auto's wiki.

I also added some new things above, like a new scoring model, how to train your own scorer, and how to make many gens from lists of prompts and seeds.

[ghost]

I used parts of your code to do some amazing stuff with it. Is your work licensed in any way? Please let me know if I can use some of your code for my custom script

[grexzen]

Please use anything you want, as long as we are all sharing, innovation never stops.

[chaewai]

This doesn't let me select the script, I get the following error on load:

Error loading script: chad_scorer.py
Traceback (most recent call last):
File "C:\SDwebUI\stable-diffusion-webui\modules\scripts.py", line 195, in load_scripts
module = script_loading.load_module(scriptfile.path)
File "C:\SDwebUI\stable-diffusion-webui\modules\script_loading.py", line 13, in load_module
exec(compiled, module.dict)
File "C:\SDwebUI\stable-diffusion-webui\scripts\chad_scorer.py", line 24, in
with open(state_name, "wb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/content/drive/MyDrive/AI/chadscorer.pth'

[grexzen]

Thanks, I'll fix it.

[ghost]

Can we get this integrated into the API :]

[AG-w]

It will be nice if it can integrate to image browser

[jelling]

Does this script still work? I added chad_scorer.py to scripts and it shows in the dropdown. But nothing different happens when I run with it enabled: