Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High packet loss with SRT streams on MacBooks #2691

Closed
saltarob opened this issue Mar 15, 2023 · 95 comments
Closed

High packet loss with SRT streams on MacBooks #2691

saltarob opened this issue Mar 15, 2023 · 95 comments
Labels
Type: Question Questions or things that require clarification
Milestone
v1.5.3

Comments

@saltarob
Copy link

Hi, after switching from my old Mac Pro 5,1 to a newer MacBook Pro 2019 I realised that my SRT stream suffered under high packet loss even if the SRT settings and the network conditions didn't change.

After testing and analysing with "lib-tcpdump-processing" I found out that on the Mac Pro only 1-3 % of the packets where retransmitted whereas on the MacBooks (I made tests with 3 MacBooks) 30 % where lost in the first transmission and had to be retransmitted. The 30 % retransmission needs of course a much bigger overhead and a big latency to be recovered.

All computers are connected with ethernet cables to the same router and the tests were made with the same video, same encoding settings, same SRT settings. But the results were consistent over many tests. Mac Pro 5,1 with 1-3 % packet loss and the MacBook Pros with up to 30% packet loss.

I made the test with different versions of OBS and FFmpeg (libsrt 1.5.1 and libsrt 1.4.1) on the sender side and with Wowza and Nimble Server on the Receiver side. I tested it with macOS 12 and macOS 13 and with Windows 10 in Bootcamp partitions.

Is there any explanation for that strange behaviour?
@saltarob saltarob added the Type: Question Questions or things that require clarification label Mar 15, 2023
@maxsharabayko
Copy link
Collaborator

Can be something to do with the network driver.
Could you test UDP sending performance using srt-xtransmit? Just replace SRT with UDP, options like latency can be omitted.
https://github.com/maxsharabayko/srt-xtransmit/blob/master/docs/metrics.md

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@ethouris
Copy link
Collaborator

SRT is a derived project there so if you specify explicitly any options for SRT, the same you should specify here. If you didn't compile SRT from source, follow specific instruction in the documentation - SSL is a special kind of dependency for Mac platform and it needs to be treated with care.

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@ethouris
Copy link
Collaborator

That one of the testing applications you can use to check if that's the problem. You can also try the same with the srt-live-transmit app that is among the demonstration applications. I don't know if it is available in the brew packages.

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@maxsharabayko
Copy link
Collaborator

@saltarob

Thank you. Unfortunately I run into some problems while compiling art-xtransmit: CMake Error at /usr/local/Cellar/cmake/3.26.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message): Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY OPENSSL_INCLUDE_DIR) Call Stack (most recent call first): /usr/local/Cellar/cmake/3.26.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE) /usr/local/Cellar/cmake/3.26.0/share/cmake/Modules/FindOpenSSL.cmake:670 (find_package_handle_standard_args) submodule/srt/CMakeLists.txt:368 (find_package) I tried to export all PATH according your instructions and the Homebrew instructions but still it is not working. How can I fix it? Thank you.

Did you do:

brew install cmake
brew install openssl
export OPENSSL_ROOT_DIR=$(brew --prefix openssl)
export OPENSSL_LIB_DIR=$(brew --prefix openssl)"/lib"
export OPENSSL_INCLUDE_DIR=$(brew --prefix openssl)"/include"

@maxsharabayko
Copy link
Collaborator

That one of the testing applications you can use to check if that's the problem. You can also try the same with the srt-live-transmit app that is among the demonstration applications. I don't know if it is available in the brew packages.

@ethouris, Please don't confuse @saltarob . srt-live-transmit does not generate UDP traffic.

@ethouris
Copy link
Collaborator

Sorry, of course. I thought he is testing on some live stream.

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@maxsharabayko
Copy link
Collaborator

Yes. I copied and pasted the commands from the GitHub page.

Well, CMake can't locate the OpenSSL library. So there is something to be done with the system paths.

In fact, encryption is not needed for these tests. It might be easier to just disable it using -DENABLE_ENCRYPTION=OFF CMake option.
See here.

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@maxsharabayko
Copy link
Collaborator

Replace %metricsfreq% with your value, e.g. 1s.

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@maxsharabayko
Copy link
Collaborator

Could you share the command line you use to run srt-xtransmit? Maybe add -v to enable verbose output.

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@maxsharabayko
Copy link
Collaborator

Try

srt-xtransmit generate "udp://127.0.0.1:4200" --enable-metrics --sendrate 5Mbps --duration 120s -v

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@ethouris
Copy link
Collaborator

I have more less the same description for this error 49: "Can't assign requested address".

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@maxsharabayko
Copy link
Collaborator

 Here is the result.

@saltarob ❓

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 •
edited
Loading

metricsfile.csv

Oh sorry the attachment didn't make it. Here the metricsfile again.

@maxsharabayko
Copy link
Collaborator

Not a single packet is received according to the file you provide.

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@maxsharabayko
Copy link
Collaborator

Hm, try

$ srt-xtransmit receive "udp://:4200" --enable-metrics

You should see the output in the terminal. Important is the number of packets detected as lost if any.

@ethouris
Copy link
Collaborator

You can also try nc (netcat) command to see if the "generate" command has actually sent any packets to the given port. Or use wireshark/tshark to record them.

@saltarob
Copy link
Author

saltarob commented Mar 16, 2023 via email

@saltarob
Copy link
Author

Okay thank you.

@maxsharabayko
Copy link
Collaborator

maxsharabayko commented Mar 17, 2023 •
edited
Loading

@saltarob

After testing and analysing with "lib-tcpdump-processing" I found out that on the Mac Pro only 1-3 % of the packets where retransmitted whereas on the MacBooks (I made tests with 3 MacBooks) 30 % where lost in the first transmission and had to be retransmitted. The 30 % retransmission needs of course a much bigger overhead and a big latency to be recovered.

To clarify, do that 30% come from Wireshark capture taken on the sender side (MacBook) or the receiver side (AWS)?

@saltarob
Copy link
Author

Both. Following OBS on MacBook Pro as Sender and Wowza on AWS as Receiver:

OBSSender

WowzaReceiverOBS

@ethouris
Copy link
Collaborator

Ok, I prepared a branch that is visible also through the #2694 draft PR.

Please get this version of SRT for this. Note that if you compiled srt-xtransmit from sources, you should simply checkout this version in the SRT submodule of srt-xtransmit. You need to add a new remote (mine - github.com/ethouris/srt) and select this branch - dev-try-sendmsg-check.

@saltarob
Copy link
Author

I am so sorry. I never made such things before. I followed the instructions from here. How do I add the new submodule?

@ethouris
Copy link
Collaborator

ethouris commented Mar 17, 2023 •
edited
Loading

No no, you don't add the submodule. It is there already, just when you are in the root directory of the srt-xtransmit repository, cd to submodule/srt. You should have had all submodules updated in order to compile srt-xtransmit, so I guess this one, too.

If you do now git remote, it should show you the "origin" named remote with the address of SRT repo.

Now do first git remote add, give it some name and the address of my repo fork (towards the address you can see, replace "Haivision" with "ethouris"). Then do "git fetch" with the name of this newly added remote. After this is done, do git checkout dev-try-sendmsg-check. Then recompile everything in srt-xtransmit repo.

@saltarob
Copy link
Author

Thank you. I don't the address with the git remote command. Then I get that error.

% cd srt
% ls
CMakeFiles		Makefile		cmake_install.cmake	haisrt.pc		srt.pc			version.h
% git remote
origin
% git remote add ethouris https://github.com/ethouris/srt/tree/dev-try-sendmsg-check
% git fetch ethouris
fatal: repository 'https://github.com/ethouris/srt/tree/dev-try-sendmsg-check/' not found

@ethouris
Copy link
Collaborator

Again: I said: use the address that you can see after displaying it by git remote and replace "Haivision" with "ethouris". Not click on the link that was actually added accidentally (I copied the branch name from the PR and it created this link).

git remote add ethouris https://github.com/ethouris/srt
git fetch ethouris
git checkout dev-try-sendmsg-check

@saltarob
Copy link
Author

Well as I said it doesn't show the address in my terminal. But I figured it out now. In my first trial I added the wrong address. Thank you.

image

@saltarob
Copy link
Author

Okay I built it again now. Show I do now the exact same test?

(receiver)
srt-xtransmit receive udp://:4200 --enable-metrics -v

(sender)
srt-xtransmit generate udp://127.0.0.1:4200 --enable-metrics --sendrate 10Mbps -v

@ethouris
Copy link
Collaborator

Try this time with SRT, and on the sender side add --loglevel note. Observe logs. You may also redirect all output to a file while observing it ( |& tee LOGFILE).

@saltarob
Copy link
Author

Thank you. Here is the log file. I hope it is okay.

LOGFILE.txt

@ethouris
Copy link
Collaborator

No errors? Strange. Do you still experience such a high packet loss in this transmission?

@saltarob
Copy link
Author

Yes. Just tested it again. OBS to Nimble Server from the MacBook:

image

@saltarob
Copy link
Author

Here the same from the desktop computer under same conditions:

image

@saltarob
Copy link
Author

Now one interesting aspect is if I do a test with Wirecast with the older libsrt 1.4.1 version from the same MacBook Pro 2019 I get a perfect transmission.

image

I tried to reproduce these results with FFmpeg libsrt 1.4.1. But with this test I got the same bad results. So I don't know what is different in Wirecast.

@saltarob
Copy link
Author

Wouldn't it be better to make the srt-xtransmit test between the MacBook and the AWS server?

@maxsharabayko
Copy link
Collaborator

Now one interesting aspect is if I do a test with Wirecast with the older libsrt 1.4.1 version from the same MacBook Pro 2019 I get a perfect transmission.

Yes, please.

@saltarob
Copy link
Author

saltarob commented Mar 20, 2023 •
edited
Loading

Okay, I tested MacBook Pro to AWS server with following commands:

srt-xtransmit generate srt://52.31.54.34:4200 --enable-metrics --sendrate 5Mbps -v --duration 60s

srt-xtransmit receive "srt://:4200" --enable-metrics -v --loglevel note |& tee LOGFILE.txt

Here the log file:

LOGFILE.txt

@saltarob
Copy link
Author

I made another test with OBS sending to srt-xtransmit on AWS. Don't know if it is useful for something. I changed the command a bit because there were to many failed checksum reports.

srt-xtransmit receive "srt://:4200" --loglevel note |& tee LOGFILE.txt

LOGFILE.txt

@saltarob
Copy link
Author

Any updates? Seems like srt-xtransmit is not helping to analyse the problem?

@saltarob
Copy link
Author

No responses anymore?

@maxsharabayko
Copy link
Collaborator

No issues, no dropped packets here.

The other one (OBS sending to srt-xtransmit on AWS) there are packet drops on the receiver and packet reordering on the network.

If those are from the same machine, then does not look like SRT problem. Consider increasing the latency to reduce drops.
Not sure why there is reordering in the second case if the network is the same.

@saltarob
Copy link
Author

Thank you. Yes same machine, same network, same bitrate. Of course I can increase the latency but 30 % of the bandwidth is gone for resending the lost packets because of the problem. What kind of problem can it be then? I am wondering why the SRT version in Wirecast can do it with under 1% lost packages and all other SRT versions have a packet loss of 10-30%. On the same machine, same bitrate, same network.

@maxsharabayko
Copy link
Collaborator

Well, it depends. I would suggest you first find what is different and try to minimize the difference to find the reason.
For example, OBS might have an old SRT version, maybe not performing that great. Or encoding issues delaying packet sending.

@saltarob
Copy link
Author

Thank you so much. Your feedback helps a lot. Since Wirecast is using the old SRT version and OBS the newest one I knew that should not be the issue. Encoding: I tried all encoders. And on other machines the same encoder is working fine.

Then I thought maybe something in the network is causing the SRT packet mess with MacBooks. So I made a test with the MacBook Pro 2019 in a different house with an independent, different network. And it worked. Packet loss under 1 %.

Now can I ask you what could cause that SRT packet mess just for certain machines (MacBooks) and certain SRT versions? Is there a way to find the cause? Are there certain known network settings which affect SRT streams negatively?

Unfortunately the problematic network is the production environment for the live streams.

Thank you.

@ethouris
Copy link
Collaborator

Hard to say. The only way to know anything more from this is to record a pcap on both sides for some time during this transmission and then analyze it. Dropped packets will be clearly visible in the pcap from non-monotonic sequences (those can be detected by some script) and the time comparison could be made. This can detect local drops or allow to determine an unusual delay if it happened on the originating machine.

@ethouris
Copy link
Collaborator

ethouris commented Oct 5, 2023

Closing due to inactivity. Please reopen if anything new appears.

@ethouris ethouris closed this as completed Oct 5, 2023
@maxsharabayko maxsharabayko added this to the v1.5.3 milestone Oct 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Question Questions or things that require clarification
Projects
None yet
Milestone
v1.5.3
Development

No branches or pull requests
3 participants
@saltarob @ethouris @maxsharabayko