guard all2all from empty transfer

[mfoerste4]

Adding guards to prevent zero sized sends/receives during NCCL all2all block. This helped to circumvent the issue observed in #722 .

[magnatelee]

@mfoerste4 looks like NCCL fixed the old issue that it required all the participants to participate the exchange; in the past I had to make sure every transfer is non-empty. glad to see that issue has been fixed.

[mfoerste4]

@mfoerste4 looks like NCCL fixed the old issue that it required all the participants to participate the exchange; in the past I had to make sure every transfer is non-empty. glad to see that issue has been fixed.

Note that this is just a workaround for the issue. NCCL should also accept zero sized messages as long as the send/recv dimensions are pairwise consistent.

[magnatelee]

Note that this is just a workaround for the issue. NCCL should also accept zero sized messages as long as the send/recv dimensions are pairwise consistent.

FWIW, I think there's a bigger question of whether an empty kernel should be allowed. You can't launch a kernel with empty launch domain in CUDA, and CUDA libraries may have chosen to be consistent with this behavior. Disallowing / under-specifying behavior of empty transfers can be one such example. I agree that it'd be certainly nicer if NCCL handled empty transfers gracefully.