Debias Estimation loss

[sdbds]

paper:https://arxiv.org/pdf/2310.08442.pdf

just change for SNR weight like min-snr-gamma

[vvern999]

It seems you forgot to add

                   if args.debiased_estimation_loss:
                        loss = apply_debiased_estimation(loss, timesteps, noise_scheduler)

in train_network.py

[sdbds]

It seems you forgot to add

                   if args.debiased_estimation_loss:
                        loss = apply_debiased_estimation(loss, timesteps, noise_scheduler)

in train_network.py

thank u for reviewing,i add it.

[kohya-ss]

Thank you for this!

I only run the test training yet, but it seems to be very promising. I will do more testing and merge this into main.

[feffy380]

@sdbds any idea what adjustment needs to be made for v_prediction and zero_terminal_snr? ZSNR in particular will break the weight calculation due to division by zero

[sdbds]

v_prediction and zero_terminal_snr

i think v_prediction and zero_terminal_snr used in SD V2 so it cant use this at the same time...

for some histroy,not all loss function implement in all script,so i think just not use this in SD V2 model.

[FurkanGozukara]

What is this and how to use with sdxl or sd 1.5 can you give some more info please?

[sdbds]

What is this and how to use with sdxl or sd 1.5 can you give some more info please?

u can think it is min-SNR plus version....

[FurkanGozukara]

What is this and how to use with sdxl or sd 1.5 can you give some more info please?

u can think it is min-SNR plus version....

sadly i don't know that too

could you give little info? and how to use?

[sdbds]

What is this and how to use with sdxl or sd 1.5 can you give some more info please?

u can think it is min-SNR plus version....

sadly i don't know that too

could you give little info? and how to use?

You can imagine it as an operation that automates the processing of noise, allowing for faster model fitting, as well as balancing out some of the color issues.

[FurkanGozukara]

What is this and how to use with sdxl or sd 1.5 can you give some more info please?

u can think it is min-SNR plus version....

sadly i don't know that too
could you give little info? and how to use?

You can imagine it as an operation that automates the processing of noise, allowing for faster model fitting, as well as balancing out some of the color issues.

thank you so much

[FurkanGozukara]

doing a SDXL DreamBooth training and testing debiased_estimation_loss right now
i hope it further improves

[FurkanGozukara]

The "debiased_estimation_loss" led to overtraining.

Interestingly, the effect of this was the amplification of a constant learning rate (LR) significantly.

Also, my U-Net combined with TE1 training is producing NaNs - exactly same settings of U-NET only

[araleza]

Hello, I don't know much about this new debiased_estimation_loss, but I wanted to ask if it was relevant to a training issue I've been seeing:

Any training images I have with large blocks of (e.g.) red and blue areas, such as someone wearing bright red clothing in front of a bright blue wall, almost immediately look terrible when Dreambooth training is performed. Other images without these blocks of colors look fine, so it doesn't seem to be due to overtraining.

It doesn't seem to be a fundamental issue with Stable Diffusion, as if I use the base SDXL model and prompt a picture that produces a person with bright red clothing in front of a bright blue wall, the image looks great. I just can't train any new images of this type with sd-scripts.

Does this effect sound related to your work?

[sdbds]

All of the current noise-related parameters have a noticeable effect on color contrast, i dont ensure if it had affect training Here are some examples of parameters that have a significant impact:

$noise_offset $multires_noise_iterations $min_snr_gamma $ip_noise_gamma $debiased_estimation_loss

Additionally, only one of these three parameters can generally be chosen for use. $min_snr_gamma $ip_noise_gamma $debiased_estimation_loss

All of the current noise-related parameters have a noticeable effect on color contrast, i dont ensure if it had affect training
Here are some examples of parameters that have a significant impact:

$noise_offset
$multires_noise_iterations
$min_snr_gamma
$ip_noise_gamma
$debiased_estimation_loss

Additionally, only one of these three parameters can generally be chosen for use.
$min_snr_gamma
$ip_noise_gamma
$debiased_estimation_loss