Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement dynamic weight formula in Advanced mode #52

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

wbclark
Copy link

@wbclark wbclark commented Aug 27, 2024

This provides the option to use dynamic mask weight to more smoothly blend the conditioning; for example, you can set the weight as =exp(-((x-0.5)**2 + (y-0.5)**2) / (2 * 0.15**2)) to give it a Gaussian distribution (bell durve) that falls off smoothly as it moves away from the center of the image, or use two regions with respective weights =y and =(1-y) to gradually transition between the prompts as you move from top to bottom along the image.

This provides the option to use dynamic mask weight to more smoothly
blend the conditioning; for example, you can set the weight as
`=exp(-((x-0.5)**2 + (y-0.5)**2) / (2 * 0.15**2))` to give it a
Gaussian distribution (bell durve) that falls off smoothly as it
moves away from the center of the image, or use two regions with
respective weights `=y` and `=(1-y)` to gradually transition between
the prompts as you move from top to bottom along the image.
@Haoming02
Copy link
Owner

Since the effect of this Extension isn't perfectly precise to begin with, doing these fancy maths feels rather... redundant.

Do you have any examples where these changes improve the outcome substantially?

@wbclark
Copy link
Author

wbclark commented Aug 27, 2024

Hi @Haoming02 , thanks for this outstanding extension, and thanks for the very quick feedback on my PR -- sorry I didn't mark it as a draft at first and provide more context just yet (I'm at the end of a long day here and I forgot that you would probably see it right away due to the time zone)

Since the effect of this Extension isn't perfectly precise to begin with, doing these fancy maths feels rather... redundant.

Yeah, I agree the Gaussian may be a bit overkill, but the linear interpolation works really nicely... I'd suggest to give a try to create a transition with styles, texture, mood, background etc, and still use the same sharply-bounded boxes for precise local character control.

What I propose to do is this:

  1. I'd like to add a 't' variable for time step / noise level, this would be extremely useful as you can control which prompt applies to more coarse or fine details of the image and conserve a lot of context this way. For example, by not applying a style prompt before it's actually needed, and dropping out a composition prompt once the base composition is set and the model's self-attention can maintain it.
  2. I have a list of more dynamic weight formulas to test and would like to add the good ones to docs/examples and consider adding the most useful ones as presets to make it easy for users to apply without any fancy maths

But I need to sleep first as it's already very late here. :P

@Haoming02 Haoming02 marked this pull request as draft September 30, 2024 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants