Why are eight network interface cards (NICs) used instead of four in a two-node, 16-GPU test setup with A100 GPUs?

[lmhahatest]

When conducting a two-node, eight-GPU test using nccltest with A100 GPUs (each with four network interfaces), why do we need to use eight network interfaces? It seems that two per machine, totaling four, would suffice. Why is the default to use eight? Please help explain, thank you.

Why are eight network interface cards (NICs) used in a two-node, 16-GPU test setup with A100 GPUs, and what does the established ring topology look like?

[kiskra-nvidia]

I don't think NCCL needs to use 8 NICs; however, if they are available and it thinks it can gain better performance that way, it will use them. You can tell NCCL which NICs (not) to use via the NCCL_IB_HCA variable -- limit it to two per node and you should see that it still works, but probably more slowly.

What benchmark were you running? Something like SendRecv indeed forms a single ring and so should use just 4 NICs. AllReduce, on the other hand, will form multiple rings so that it can utilize all the available NICs for maximum bandwidth.

[lmhahatest]

o,thanks very much , use nccl-tests.

When using two machines with four GPUs and four NICs (200Gb/s) for the A100, the busbw is 78. When using two machines with four GPUs and two NICs (200Gb/s), the busbw is 45. Assuming the use of two machines with eight GPUs and eight NICs (400Gb/s), can the busbw reach 400GB/s * 80% ?