Fix gradients synchronization for multi-GPUs training

[Isotr0py]

• Related issue: About GPU utilization #965

• Fix [Bug] Gradients not synchronized  #924

• Remove train_util.transform_if_model_is_DDP, this caused the gradients sync bug on DDP. Now model will be unwrapped by accelerator.unwrap_model() manually.

• Clean accelerator.prepare() codes in train_network.py

• Sync lora network gradients in train_network.py manually.

For train_network.py, since we apply the lora through replacing the forward() method instead of the module, the DDP lora won't sync gradients automatically when calling accelerator.backward(loss). Hence we need to use accelerator.reduce() to sync grad manually.

[Isotr0py]

Gradients and parameters after all_reduced should be synchronized now.

[kohya-ss]

Thank you for this! I didn't test multi-gpu training, but the PR seems to be very important.

[deepdelirious]

This seems to have broken sampling -

Traceback (most recent call last):
  File "/workspace/train.py", line 44, in <module>
    train(args)
  File "/workspace/sdxl_train.py", line 610, in train
    sdxl_train_util.sample_images(
  File "/workspace/library/sdxl_train_util.py", line 367, in sample_images
    return train_util.sample_images_common(SdxlStableDiffusionLongPromptWeightingPipeline, *args, **kwargs)
  File "/workspace/library/train_util.py", line 4788, in sample_images_common
    latents = pipeline(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/workspace/library/sdxl_lpw_stable_diffusion.py", line 926, in __call__
    dtype = self.unet.dtype
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'DistributedDataParallel' object has no attribute 'dtype'

[kohya-ss]

@deepdelirious I've updated the dev branch. I didn't test with multiple GPUs, but I think it will fix the sampling.

[FurkanGozukara]

I dont see these in code where ?

ddp_bucket_view
ddp_gradient_as_bucket_view