Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement samplers correctly #2

Open
1 of 4 tasks
152334H opened this issue Feb 4, 2023 · 10 comments
Open
1 of 4 tasks

Implement samplers correctly #2

152334H opened this issue Feb 4, 2023 · 10 comments
Labels
enhancement New feature or request speed related to speed improvements

Comments

@152334H
Copy link
Owner

152334H commented Feb 4, 2023

  • rudimentary dpm++2m implementation
  • explore other DPM-Solver samplers
  • figure out if k-diffusion is still possible
  • UniPC
@152334H 152334H closed this as completed Feb 5, 2023
@Ryu1845
Copy link
Collaborator

Ryu1845 commented Feb 15, 2023

@152334H thoughts about https://github.com/wl-zhao/UniPC ?

@Ryu1845
Copy link
Collaborator

Ryu1845 commented Feb 15, 2023

Examples on images AUTOMATIC1111/stable-diffusion-webui#7710

@152334H
Copy link
Owner Author

152334H commented Feb 15, 2023

Their project says they support $\epsilon_\theta(x_t, t)$ models, so I'll give it a go

Their code in https://github.com/wl-zhao/UniPC/blob/main/example/stable-diffusion/ldm/models/diffusion/uni_pc/uni_pc.py also seems very similar to the DPM-Solver repo, which i'll be integrating soon, so that's good

@152334H
Copy link
Owner Author

152334H commented Feb 15, 2023

on a related note, I realised a few days ago (thanks to mrq) that my implementation of k-diffusion was actually completely wrong.

I'll be adding code that actually runs dpm++2m correctly in about an hour (the K diffusion integration is most likely screwed), then I can go for uniPC

@152334H 152334H reopened this Feb 15, 2023
@152334H
Copy link
Owner Author

152334H commented Feb 15, 2023

I'll write a larger blog about this later, but to clarify, this is what happened:

  • I added functions for k-diffusion, but didn't actually call on them in my code
  • All "DPM++2M" results before today's commit were actually done with p_sample, which is basically random sampling of the gaussian distribution with the mean and variance values calculated by the model at each step. Yes, this means that the "really good 10 step results" were actually just plain DDIM.
  • After realising this, I tried to fix it. As it turns out, integrating the k-diffusion library into TorToiSe's diffusion model is actually a non-trivial task, because k-diffusion defines its own sigmas that conflict with the beta scheduler that's used to calculate $x_\theta(x_t,t)$ (which k-diffusion expects) in p_mean_variance. I tried to calculate the x0 prediction from the raw model output + karras' sigmas, but I got a bunch of noise. It's entirely possible I just failed to write the right integration code for the k-diffusion samplers, but I'm not going to try working on it for now.
  • I have instead opted to make use of the $\epsilon_\theta(x_t, t)$ outputs from the diffusion model directly, discarding the $\Sigma_\theta(x_t, t)$ prediction, and using it with the DPM-Solver repo to run DPM++2M. This "worked", but doesn't produce good results for steps<=20, and I assume this happens because of the discarded variance || because I wrote a wrong constant somewhere.

tldr: past samplers were fake; dpm++2m is now experimental but real, DDIM+cond_free is preferable for steps < 20 until better samplers exist.

Consequently, I'm making DDIM the default sampler for ultra_fast for now, and have created a new preset (very_fast) that uses DPM++2M with more steps.

All claims stated here only apply to fp32 inference; I have no idea what the results are like on --half yet.

@152334H 152334H changed the title dpm++ samplers Implement samplers correctly Feb 15, 2023
@152334H 152334H added enhancement New feature or request speed related to speed improvements labels Feb 15, 2023
@diichen
Copy link

diichen commented Jul 12, 2024

It seems you should:

  1. use GaussianDiffusion instead of SpacedDiffusion.
  2. use discrete NoiseScheduleVP instead of continuous NoiseScheduleVP.

@152334H
Copy link
Owner Author

152334H commented Jul 12, 2024

well. yes. a bit late for that now though

@diichen
Copy link

diichen commented Jul 12, 2024

Have you tried any other solver or Rectified Flow for diffusion inference speed up?

@152334H
Copy link
Owner Author

152334H commented Jul 12, 2024

I don't actually work on this project anymore

@diichen
Copy link

diichen commented Jul 12, 2024

ok, Thanks for your work, it helps me a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request speed related to speed improvements
Projects
None yet
Development

No branches or pull requests

3 participants