Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix baddbmm handling of "beta" special-case #1801

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Birch-san
Copy link

baddbmm() has a beta coefficient, by which bias may be multiplied.

in the special-case where beta=0: bias may be ignored (confirmed by docs):

If beta is 0, then input will be ignored, and nan and inf in it will not be propagated.

we use this in diffusers / stable-diffusion to avoid adding attention bias when none is specified:
https://github.com/huggingface/diffusers/blob/bbab8553224d12f7fd58b0e65b0daf899769ef0b/src/diffusers/models/cross_attention.py#L237

currently, the special-case is determined by a comparison with 1.0 rather than 0.0. it looks like it was copied from alpha's special-case:

if beta.val != 1.0:
# Apply scaling factor beta to the bias.
bias = mb.mul(x=beta, y=bias, name=bias.name + "_scaled")
context.add(bias)
if alpha.val != 1.0:
# Apply scaling factor alpha to the input.
batch1 = mb.mul(x=alpha, y=batch1, name=batch1.name + "_scaled")
context.add(batch1)

changing this, fixed compilation of diffusers' UNetCondition2D model for me (which employs that baddbmm in CrossAttnProcessor).

@aseemw
Copy link
Collaborator

aseemw commented Mar 12, 2023

Thanks for the PR, can you please add a unit test as well in the torch unit test file, which fails without this change, but passes with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants