Skip to content
This repository has been archived by the owner on Oct 31, 2023. It is now read-only.

A warning appears in DDP using ConvNeXt #142

Open
Toshimasa-Matsumoto opened this issue May 12, 2023 · 1 comment
Open

A warning appears in DDP using ConvNeXt #142

Toshimasa-Matsumoto opened this issue May 12, 2023 · 1 comment

Comments

@Toshimasa-Matsumoto
Copy link

When I start training using DDP with ConvNeXt, the following warning appears.

What caused the warning?
Also, is there a way to avoid the warning?

UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed.  This is not an error, but may impair performance.
grad.sizes() = [1024, 1, 7, 7], strides() = [49, 1, 7, 1]
bucket_view.sizes() = [1024, 1, 7, 7], strides() = [49, 49, 7, 1] (Triggered internally at ../torch/csrc/distributed/c10d/reducer.cpp:323.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass

Thank you.

@anhtienng
Copy link

anhtienng commented Jun 26, 2023

I also got this warning.
I found this https://github.com/pytorch/pytorch/issues/47163. So the rule is you need add .contiguous() after the permute operation
image
image

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants