Add support for different model prediction types in DDIMInverseScheduler #2619

clarencechen · 2023-03-09T09:22:17Z

Resolve alpha_prod_t_prev index issue for final step of inversion
@patrickvonplaten TODO: Update numerical tests to reflect two more steps of inversion

HuggingFaceDocBuilderDev · 2023-03-09T09:35:30Z

The documentation is not available anymore as the PR was closed or merged.

Resolve alpha_prod_t_prev index issue for final step of inversion

src/diffusers/schedulers/scheduling_ddim_inverse.py

…improve-ddim-inverse

… old kwarg

Derive predicted noise from model output before dist. regularization Style cleanup

This reverts commit 76ea9c8.

…improve-ddim-inverse

clarencechen · 2023-03-12T23:20:21Z

@patrickvonplaten
Here is a notebook demonstrating a working run of the Pix2PixZero pipeline with Stable Diffusion 2. I find that the value of cross_attention_guidance_amount needs to be lowered to 0.01 or less in order to prevent loss explosion.

https://colab.research.google.com/gist/clarencechen/4166c7d1ad29b09940a13f62437de634/ddiminversetesting.ipynb

sayakpaul · 2023-03-13T05:35:13Z

Thanks so much!

What's pending in this PR?

Maybe we should mention

I find that the value of cross_attention_guidance_amount needs to be lowered to 0.01 or less in order to prevent loss explosion.

in the Pix2PixZero docs?

src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_pix2pix_zero.py

src/diffusers/schedulers/scheduling_ddim_inverse.py

sayakpaul

Thanks a lot for working on this!

I see some non-trivial changes in the scheduler script. Do you mind elaborating a bit more on those?

clarencechen · 2023-03-13T07:01:01Z

Thanks so much!

What's pending in this PR?

Maybe we should mention

I find that the value of cross_attention_guidance_amount needs to be lowered to 0.01 or less in order to prevent loss explosion.

in the Pix2PixZero docs?

@sayakpaul I just need to add new tests for Pix2Pix Zero with Stable Diffusion V2, which require uploading target images that are slightly different from what exists for current tests, but which are semantically and stylistically nearly identical.

sayakpaul · 2023-03-13T07:03:43Z

Sure, thanks for letting us know! Let us know if we can help in unblocking anything for you here.

…improve-ddim-inverse

clarencechen · 2023-03-14T01:02:54Z

@sayakpaul @patrickvonplaten In order to upload expected results for a Pix2PixZero slow test with SD 2, I need to update the following repo:
https://huggingface.co/datasets/hf-internal-testing/diffusers-images
I created a new local branch with the new data, but I need push access to the hf-internal-testing org in order to push the branch to remote and open a PR there.

src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_pix2pix_zero.py

tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py

sayakpaul · 2023-03-14T09:00:05Z

tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py

+    def test_stable_diffusion_2_pix2pix_full(self):
+        # numpy array of https://huggingface.co/datasets/hf-internal-testing/diffusers-images/blob/main/pix2pix/dog_2.png
+        expected_image = load_numpy(
+            "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/pix2pix/dog_2.npy"


I believe this needs to be in https://huggingface.co/datasets/hf-internal-testing/diffusers-images/tree/main/pix2pix? Feel free to open a PR to the repository to add it.

Indeed, could you maybe open a PR to add the numpy image? This would be very nice!

https://huggingface.co/datasets/hf-internal-testing/diffusers-images/discussions/10

Can't currently publish due to rate limiting

Thanks a lot for opening it - merged it!

sayakpaul

I think we're in a very good shape to merge this. Thanks so much for this important PR :)

sayakpaul · 2023-03-14T09:01:58Z

@sayakpaul @patrickvonplaten In order to upload expected results for a Pix2PixZero slow test with SD 2, I need to update the following repo: https://huggingface.co/datasets/hf-internal-testing/diffusers-images I created a new local branch with the new data, but I need push access to the hf-internal-testing org in order to push the branch to remote and open a PR there.

Yeah feel free to open a PR to https://huggingface.co/datasets/hf-internal-testing/diffusers-images following instructions from https://huggingface.co/datasets/hf-internal-testing/diffusers-images/discussions (Click on "New pull request" and follow the instructions).

tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py

…_zero.py

patrickvonplaten · 2023-03-14T20:25:07Z

Great work @clarencechen - thanks!

…ler (huggingface#2619) * Add support for different model prediction types in DDIMInverseScheduler Resolve alpha_prod_t_prev index issue for final step of inversion * Fix old bug introduced when prediction type is "sample" * Add support for sample clipping for numerical stability and deprecate old kwarg * Detach sample, alphas, betas Derive predicted noise from model output before dist. regularization Style cleanup * Log loss for debugging * Revert "Log loss for debugging" This reverts commit 76ea9c8. * Add comments * Add inversion equivalence test * Add expected data for Pix2PixZero pipeline tests with SD 2 * Update tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py * Remove cruft and add more explanatory comments --------- Co-authored-by: Patrick von Platen <[email protected]>

clarencechen mentioned this pull request Mar 9, 2023

I don't get DDIMInverseScheduler working #2577

Closed

clarencechen force-pushed the improve-ddim-inverse branch from 00bab47 to fcee91a Compare March 9, 2023 09:31

Add support for different model prediction types in DDIMInverseScheduler

3d93466

Resolve alpha_prod_t_prev index issue for final step of inversion

clarencechen force-pushed the improve-ddim-inverse branch from 2b7a9dc to 3d93466 Compare March 9, 2023 20:17

Fix old bug introduced when prediction type is "sample"

64ed3ea

patrickvonplaten reviewed Mar 10, 2023

View reviewed changes

src/diffusers/schedulers/scheduling_ddim_inverse.py Show resolved Hide resolved

clarencechen added 2 commits March 10, 2023 15:13

Merge branch 'main' of https://github.com/huggingface/diffusers into …

5228f55

…improve-ddim-inverse

Add support for sample clipping for numerical stability and deprecate…

e5f5b4a

… old kwarg

clarencechen force-pushed the improve-ddim-inverse branch 5 times, most recently from 37f3096 to 01747ff Compare March 12, 2023 20:01

Detach sample, alphas, betas

1a0c13c

Derive predicted noise from model output before dist. regularization Style cleanup

clarencechen force-pushed the improve-ddim-inverse branch from 01747ff to 1a0c13c Compare March 12, 2023 20:12

clarencechen added 3 commits March 12, 2023 15:04

Log loss for debugging

76ea9c8

Revert "Log loss for debugging"

02d20ac

This reverts commit 76ea9c8.

Merge branch 'main' of https://github.com/huggingface/diffusers into …

baacb6c

…improve-ddim-inverse

sayakpaul reviewed Mar 13, 2023

View reviewed changes

src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_pix2pix_zero.py Show resolved Hide resolved

sayakpaul reviewed Mar 13, 2023

View reviewed changes

src/diffusers/schedulers/scheduling_ddim_inverse.py Outdated Show resolved Hide resolved

sayakpaul reviewed Mar 13, 2023

View reviewed changes

sayakpaul requested a review from patrickvonplaten March 13, 2023 05:48

clarencechen added 3 commits March 13, 2023 00:49

Add comments

63784fb

Merge branch 'main' of https://github.com/huggingface/diffusers into …

1d7f076

…improve-ddim-inverse

Add inversion equivalence test

9d1060e

clarencechen added 2 commits March 13, 2023 17:57

Add expected data for Pix2PixZero pipeline tests with SD 2

022faac

Merge branch 'main' of https://github.com/huggingface/diffusers into …

f029a4e

…improve-ddim-inverse

clarencechen force-pushed the improve-ddim-inverse branch from 256042e to f029a4e Compare March 14, 2023 00:59

clarencechen requested review from sayakpaul and removed request for patrickvonplaten March 14, 2023 01:03

sayakpaul reviewed Mar 14, 2023

View reviewed changes

src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_pix2pix_zero.py Show resolved Hide resolved

sayakpaul reviewed Mar 14, 2023

View reviewed changes

tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py Outdated Show resolved Hide resolved

sayakpaul reviewed Mar 14, 2023

View reviewed changes

tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py Outdated Show resolved Hide resolved

sayakpaul reviewed Mar 14, 2023

View reviewed changes

sayakpaul approved these changes Mar 14, 2023

View reviewed changes

patrickvonplaten reviewed Mar 14, 2023

View reviewed changes

tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Mar 14, 2023

View reviewed changes

tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py Outdated Show resolved Hide resolved

patrickvonplaten and others added 3 commits March 14, 2023 20:33

Update tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix…

c4054ac

…_zero.py

Merge branch 'main' into improve-ddim-inverse

71eff1d

Remove cruft and add more explanatory comments

c2857fc

patrickvonplaten merged commit ee71d9d into huggingface:main Mar 14, 2023

clarencechen deleted the improve-ddim-inverse branch March 14, 2023 20:43

patrickvonplaten mentioned this pull request May 17, 2023

[WIP] Allow DDIMInverseScheduler to use same number of noising and denoising steps #3436

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for different model prediction types in DDIMInverseScheduler #2619

Add support for different model prediction types in DDIMInverseScheduler #2619

clarencechen commented Mar 9, 2023

HuggingFaceDocBuilderDev commented Mar 9, 2023 •

edited

Loading

clarencechen commented Mar 12, 2023

sayakpaul commented Mar 13, 2023

sayakpaul left a comment

clarencechen commented Mar 13, 2023

sayakpaul commented Mar 13, 2023

clarencechen commented Mar 14, 2023 •

edited

Loading

sayakpaul Mar 14, 2023

patrickvonplaten Mar 14, 2023

clarencechen Mar 14, 2023

patrickvonplaten Mar 14, 2023

sayakpaul left a comment •

edited

Loading

sayakpaul commented Mar 14, 2023

patrickvonplaten commented Mar 14, 2023

Add support for different model prediction types in DDIMInverseScheduler #2619

Add support for different model prediction types in DDIMInverseScheduler #2619

Conversation

clarencechen commented Mar 9, 2023

HuggingFaceDocBuilderDev commented Mar 9, 2023 • edited Loading

clarencechen commented Mar 12, 2023

sayakpaul commented Mar 13, 2023

sayakpaul left a comment

Choose a reason for hiding this comment

clarencechen commented Mar 13, 2023

sayakpaul commented Mar 13, 2023

clarencechen commented Mar 14, 2023 • edited Loading

sayakpaul Mar 14, 2023

Choose a reason for hiding this comment

patrickvonplaten Mar 14, 2023

Choose a reason for hiding this comment

clarencechen Mar 14, 2023

Choose a reason for hiding this comment

patrickvonplaten Mar 14, 2023

Choose a reason for hiding this comment

sayakpaul left a comment • edited Loading

Choose a reason for hiding this comment

sayakpaul commented Mar 14, 2023

patrickvonplaten commented Mar 14, 2023

HuggingFaceDocBuilderDev commented Mar 9, 2023 •

edited

Loading

clarencechen commented Mar 14, 2023 •

edited

Loading

sayakpaul left a comment •

edited

Loading