how overall throughout calculate about all2all

[ltm920716]

Hi，
I have six H100 nodes，and each with 8*400Gb cx7 nics. And for RDMA, I use RoCE. I want to see the overall throughout.

about allreduce, it seems that the params effect little，and the busbw is the overall throughout？

abou all2all，the params effect large，as follows：

and for all2all，the busbw is for single node or something else？How can I calculate the overall throughout？I can not understand deeply about the busbw for all2all，and what params are the best to test alltoall？the performence will down with the same config when add more node

thanks！

[sjeaugey]

All numbers look good. Alltoall cannot aggregate the bandwidth of multiple NICs, so the performance you should see is the performance of a single NIC.

[ltm920716]

so how should I calculate the single nic performance？or could you recommend me a link to understand the calculation formula？