Datagram lost Increase with Decreasing its length

[PraptiSarker02]

I am using iperf3.12.0 on windows. I know iperf3 isn't officially supported on windows, but my question will be more of a conceptual rather than technical, so os shouldn't matter.

iperf command used: iperf3 -c 192.168.1.93 -t 15 -w4M -u -b 1G -V

iperf command used: iperf3 -c 192.168.1.93 -t 15 -w4M -l 8000 -u -b 1G -V

Can someone explain why does the lost and bitrate decreases when datagram payload length is increased from 1460 byte to 8000 byte?

One further question, how can the block/datagram length be 131072 byte when doing tcp test, where the maximum theoretical value is around 65535 bytes?(see image below)

iperf command used: iperf3 -c 192.168.1.93 -t 15 -w4M -V

[bmah888]

I can answer the second question...the reason you can have 128KB blocks with TCP tests is that TCP is a stream-oriented protocol. It doesn't keep the boundaries between blocks (which are basically just chunks of memory passed to a send() system call). So the TCP protocol implementation is free to break up that 128KB send into IP packets of some appropriate size to be transmitted across the network (looks like each packet holds a maximum of 1460 payload bytes, which is typical for Ethernet networks).

UDP operation is different, every "block" sent corresponds to a UDP datagram, which corresponds to an IP packet. So the block size of a UDP test has to be capable of fitting into a single IP packet. (There might be some fragmentation and reassembly of packets in the IP layer but that's transparent to the sending and receiving iperf3 processes.)

[PraptiSarker02]

Thanks, @bmah888. Can you please confirm if I understood correctly:

When transmitting through TCP protocol, that 128KB are broken down into chunks of 1460bytes and these 1460byte+ip header+tcp headers are combined to make packets, which are transmitted in sequence. In both TCP and UDP, the packet size(including overheads) is 1500Byte(at least in iperf3 default). The difference in packet is that there are no TCP-specific data, such as sequence number and acknowledgment number, in the UDP datagram/packet.

And, also would like to know about the bitrate and datagram drop with a lower datagram size(first question). Therefore, not marking your reply as the answer.

[PraptiSarker02]

@bmah888, and @davidBar-On, Is the affect of packet size on bitrate and lost an issue or it is me understanding something wrong?

[davidBar-On]

.. Is the affect of packet size on bitrate and lost an issue or it is me understanding something wrong?

The lost UDP with -l 1460 is probably an issue, but it is not clear whether it is an iperf3 issue or system/network issue. Also, the 49% lost is suspicious because it is almost exactly 50%. Following are some tests you can perform to further evaluate the issue:

1. In discussion What does "-d" debug flag have on the UDP test? #1424 you run TCP test without a router. Can you repeat the UDP test using this configuration? That will show whether the the router is causing the lost UDP packets.

2. Run the UDP test using different packets lengths, to try to find what is the packets lengths range where the problem happens. E.g., try with -l of 512, 1200, 1500, 4000.

3. Run the server with --debug=3. When a packet is lost, an "OUT OF ORDER" message is printed when the next packet arrives. This may show for example if exactly each second packet is lost. Note that probably only the beginning of the "OUT OF ORDER" messages will be relevant, as printing will reduce the server's performance, so it is expected that this will cause more packets to be lost.

[PraptiSarker02]

@davidBar-On Thank a lot for getting back. I just tried what you have asked for.

All the following result are performed without a router, it was a one-to-one connection:

1)iperf command used: iperf3 -c 192.168.0.2 -t 15 -w4M -u -b 1G -V

2a) iperf command used: iperf3 -c 192.168.0.2 -t 15 -w4M -u -b 1G -l 512 -V

2b) iperf command used: iperf3 -c 192.168.0.2 -t 15 -w4M -u -b 1G -l 1200 -V

2c) iperf command used: iperf3 -c 192.168.0.2 -t 15 -w4M -u -b 1G -l 2000 -V

2d) iperf command used: iperf3 -c 192.168.0.2 -t 15 -w4M -u -b 1G -l 4000 -V

3. iperf command used: server->[iperf3 -s --debug=3] ; client-> [iperf3 -c 192.168.0.2 -t 15 -w4M -u -b 1G -V]

-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
get_parameters:
{
        "udp":  true,
        "omit": 0,
        "time": 15,
        "num":  0,
        "blockcount":   0,
        "parallel":     1,
        "window":       4194304,
        "len":  1460,
        "bandwidth":    1000000000,
        "pacing_timer": 1000,
        "client_version":       "3.12"
}
Accepted connection from 192.168.0.1, port 49754
SNDBUF is 4194304, expecting 4194304
RCVBUF is 4194304, expecting 4194304
Setting application pacing to 125000000
[  5] local 192.168.0.2 port 5201 connected to 192.168.0.1 port 58078
interval_len 1.000068 bytes_transferred 46248420
interval forces keep
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  44.1 MBytes   370 Mbits/sec  0.012 ms  26715/58392 (46%)
interval_len 0.999942 bytes_transferred 46984260
interval forces keep
[  5]   1.00-2.00   sec  44.8 MBytes   376 Mbits/sec  0.014 ms  31689/63870 (50%)
interval_len 1.000048 bytes_transferred 47750760
interval forces keep
[  5]   2.00-3.00   sec  45.5 MBytes   382 Mbits/sec  0.014 ms  32026/64732 (49%)
interval_len 0.999995 bytes_transferred 47260200
interval forces keep
[  5]   3.00-4.00   sec  45.1 MBytes   378 Mbits/sec  0.015 ms  32196/64566 (50%)
interval_len 0.999975 bytes_transferred 47193040
interval forces keep
[  5]   4.00-5.00   sec  45.0 MBytes   378 Mbits/sec  0.010 ms  32113/64437 (50%)
interval_len 1.000015 bytes_transferred 47229540
interval forces keep
[  5]   5.00-6.00   sec  45.0 MBytes   378 Mbits/sec  0.016 ms  31992/64341 (50%)
interval_len 0.999988 bytes_transferred 47092300
interval forces keep
[  5]   6.00-7.00   sec  44.9 MBytes   377 Mbits/sec  0.014 ms  32466/64721 (50%)
interval_len 1.000022 bytes_transferred 49999160
interval forces keep
[  5]   7.00-8.00   sec  47.7 MBytes   400 Mbits/sec  0.011 ms  30108/64354 (47%)
interval_len 0.999979 bytes_transferred 47185740
interval forces keep
[  5]   8.00-9.00   sec  45.0 MBytes   377 Mbits/sec  0.011 ms  32286/64605 (50%)
interval_len 1.000016 bytes_transferred 47195960
interval forces keep
[  5]   9.00-10.00  sec  45.0 MBytes   378 Mbits/sec  0.013 ms  32107/64433 (50%)
interval_len 0.999990 bytes_transferred 46921480
interval forces keep
[  5]  10.00-11.00  sec  44.7 MBytes   375 Mbits/sec  0.017 ms  32275/64413 (50%)
interval_len 1.000005 bytes_transferred 47309840
interval forces keep
[  5]  11.00-12.00  sec  45.1 MBytes   378 Mbits/sec  0.015 ms  32024/64428 (50%)
interval_len 0.999992 bytes_transferred 47609140
interval forces keep
[  5]  12.00-13.00  sec  45.4 MBytes   381 Mbits/sec  0.018 ms  32045/64654 (50%)
interval_len 1.000012 bytes_transferred 47739080
interval forces keep
[  5]  13.00-14.00  sec  45.5 MBytes   382 Mbits/sec  0.011 ms  31875/64573 (49%)
interval_len 0.999982 bytes_transferred 46985720
interval forces keep
[  5]  14.00-15.00  sec  44.8 MBytes   376 Mbits/sec  0.009 ms  32498/64680 (50%)
interval_len 0.006160 bytes_transferred 172280
interval forces keep
[  5]  15.00-15.01  sec   168 KBytes   224 Mbits/sec  0.017 ms  122/240 (51%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-15.01  sec   678 MBytes   379 Mbits/sec  0.017 ms  474537/961439 (49%)  receiver

get_results
{
        "cpu_util_total":       76.682939818156825,
        "cpu_util_user":        22.131716565330493,
        "cpu_util_system":      54.551223252826333,
        "sender_has_retransmits":       0,
        "streams":      [{
                        "id":   1,
                        "bytes":        1412263840,
                        "retransmits":  1.8446744073709552e+19,
                        "jitter":       0,
                        "errors":       0,
                        "packets":      967304,
                        "start_time":   0,
                        "end_time":     15.000832
                }]
}
send_results
{
        "cpu_util_total":       54.87044869819627,
        "cpu_util_user":        19.885039946006518,
        "cpu_util_system":      34.985408752189755,
        "sender_has_retransmits":       18446744073709551615,
        "streams":      [{
                        "id":   1,
                        "bytes":        710876920,
                        "retransmits":  18446744073709551615,
                        "jitter":       1.6725103228936978e-05,
                        "errors":       474537,
                        "packets":      961439,
                        "start_time":   0,
                        "end_time":     15.006189
                }]
}

[davidBar-On]

@PraptiSarker02, thanks for the information you provided. I still don't understand the reason for the lost packets ... My current guess is that it is related to the Cygwin/Windows UDP implementation, and for some reason every 2nd packet "send" fails or transmits no data. Currently, if no data is transmitted, iperf3 still increases the packet count, so on the server side it would be seem as packets are lost, although no data is really lost (see PR #1380 with a proposed solution for this issue).

The following information can help to further evaluate the issue:

1. Generate a .pcap file (similar to the file you provided in Are fragmented packets handle differently in UDP Test? #1426) on the server side for -l 1460 (the default). If my guess is correct, using Wireshark we will see that no data is really lost. I.e. only empty iperf3 packets are "lost". Note that since the packets lost seem to start immediately when the test starts, then the test can run for only 2 seconds (to have smaller .pcap file)
2. If the above shows that data is lost (i.e. my guess is incorrect), then generate the .pcap file on the client side. Will allow to see if the packets are lost already on the client side.
3. Another guess I have is that if a packet send immediately follows a packet send, the the second send fails. This is is caused because of the sending "burst" (I will explain that in What does "-d" debug flag have on the UDP test? #1424). If this is correct, then with no packets burst no packet will be lost (although the throughput will be lower). To test that, set -b 1G\1.
4. The Windows CPU load during the client test may also be an important factor. It will help to get the CPU loads with -l 460 and -l 8000.

[PraptiSarker02]

@davidBar-On Thank you for helping me out in understanding networking throughput testing and how iperf3 actually works. So, I tried what you have suggested.

Instead of running it for 2 second, I actually ran it until 600000 datagrams are send, and the .pcap files are interesting, so from client side all 600000 datagram were sent and all those datagram seems to be received on server side. And, none of the datagram aren't actually empty, but still there is a 17% loss on server side.
Server Pcap.zip
Client Pcap.zip

Iperf command used: iperf3 -c 192.168.0.2 -w4M -u -b1G -k600000 -V

And, decreasing the packet number in single burst have actually worse effect on loss:
Iperf command used: iperf3 -c 192.168.0.2 -w4M -u -b1G\1 -k600000 -V

[PraptiSarker02]

I just observe that the total datagram sent don't match the total datagram received, although the wireshark says that it have received. So, my assumption is that the datagram which was not reported on was due to overhead processing of iperf. But, the question arise here is: Then the lost datagram count is just the number of corrupted data that was received, based on the UDP checksum bytes?

[davidBar-On]

I just observe that the total datagram sent don't match the total datagram received, although the wireshark says that it have received. So, my assumption is that the datagram which was not reported on was due to overhead processing of iperf. But, the question arise here is: Then the lost datagram count is just the number of corrupted data that was received, based on the UDP checksum bytes?

If you named the pcap files correctly, then the server got all the 600K packets and the missing packets are reported on the client side... That can happen when for different reasons, usually CPU or network load, Wireshark is not able to capture all packets. I assume this is the case here and it may be that the client's CPU is overloaded during the test.

Instead of running it for 2 second, I actually ran it until 600000 datagrams are send .... there is a 17% loss on server side.

As the server received all the 600K packets, it seems that the 17% lost may in the iperf3 server side (and not the client as I thought before). To evaluate that it will really help if you can provide the client and server pcap files for the 49% lost case.

In addition, as I wrote before, it will help if you will run another test with setting --debug=3 for the server. The server will list "OUT OF ORDER" for the missing packets and this will help evaluation.

[PraptiSarker02]

There might be regular system routine running at the background but not a single app, that I was running intentionally, other than wireshark. Which was also very recent.

[PraptiSarker02]

@davidBar-On need a suggestion. Now that, I am getting 5-17% lost with 1460(default) payload size is there any drawback of using 8000byte as payload for upd test? Cause the lost drop down to 0%, and the bitrate is also close to what I get from tcp test. You have already explain that decreasing the payload size increases the system overhead in discussion#1424, but now I am wondering if it is okay to use higher payload to get bitrate similar to tcp test throughput output.

[davidBar-On]

I don't see any problem in using 8KB packets for the UDP test. The default 1460 is to make sure there is no fragmentation by default, but if the netwok supports fragmentation is supported this is not an issue. Usually the packet size used depends on the network usage that is tested. E.g., if the applications that will be used in a product are sending 1K UDP packets, then iperf3 test will be with 1K packets. If the apps send 10K packets then the test will be using 10K packets, etc.

[PraptiSarker02]

Just one last question: in wifi udp test if the bitlimit "-b" is set to very high value then it is on that specific wifi mode, what is the expected consequence? i.e. for 802.11g the theoretical throughput is 54Mbps but If have "-b 500M" should I expect lost for that?

[davidBar-On]

For TCP, setting such limit will not have any effect - throughput limit is relevant to TCP only if its value is less than the maximum.

For UDP, except for the -b value, the only other limits are the sending computer resources limit - CPU, memory, etc. Therefore, it the test you describe it is expected that most of the packets that the client will send will be lost. Note that throughput for -b 500M may be worth then for -b 100M (both above the network throughput), because of overloading the sending computer resources.