iperf3 generating traffic slightly lower on one machine

[77yu77]

Hi,the scenario requires me to use iperf3 to generate traffic. And on the same machine (such as between containers), I wrote a program to send traffic to compare with iperf3. When one CPU is fully occupied, the average bandwidth is 28.35Gbits/s, while the iperf3 test shows 25.7Gbit/s with the same conditions. Does this mean that iperf3 may have some extra CPU resource consumption that causes the bandwidth to be lower?

[bmah888]

What version of iperf3 and what command-line arguments (both client and server)?

Was this result repeatable over multiple runs?

[77yu77]

Thanks you very much.

[davidBar-On]

😄
The reason that the -l impact the performance is actually as you suggested in the PR description: "Does this mean that iperf3 may have some extra CPU resource consumption that causes the bandwidth to be lower?". iperf3 has a CPU overhead before each write() and each call to write has its overhead, so if the message length is smaller more writes are performed.

[77yu77]

Yes, so when running locally, the data is read and written in memory, so the transmission speed is faster, resulting in this part of the overhead being very obvious. When transmitting across machines, the bottleneck is the network card and link, so the TCP buffer has little effect (when the network card and link are normal)? I tried my best to modify the code but could not reach 10Gbits/s for 10 Gbps network card.

[davidBar-On]

When transmitting across machines, the bottleneck is the network card and link, so the TCP buffer has little effect (when the network card and link are normal)?

The actual machine to machine permanence depends on many different parameters, including the ones you mentioned, so usually it is not possible to understand the behavior without analyzing the traffic between the different components. Different buffers sizes, Switches between the machines, etc. also impact the performance.

If your NIC supports any TCP offload engine methods (e.g. GRO) that may help. With iperf3 you can also try the Zero-copy option (-Z) to see if that method helps.

[77yu77]

Thanks! I will have a try.