Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RTMP live stream disconnects suddenly #1775

Open
cuenta0gow opened this issue Oct 11, 2024 · 2 comments
Open

RTMP live stream disconnects suddenly #1775

cuenta0gow opened this issue Oct 11, 2024 · 2 comments

Comments

@cuenta0gow
Copy link

cuenta0gow commented Oct 11, 2024 •
edited
Loading

We are having a problem with our nginx RTMP stream. The live stream works fine, but it can suddenly stop and we aren't being able to find out why after an exhausting reseach. We are using a GoPro Hero 12.

This can't be a network or camera issue, we already dismissed that. To dimiss that we set up a YouTube live stream simultaneously to verify that when our RTMP stream falls the camera still streams in YouTube. We are sure this has to be a server side issue.

The stream falls every single day at the same time, around 10:20-10:40 and then at 14:00. We checked everything, but we found nothing. The server is a clean server fully dedicated to RTMP live streaming.

This is our simple RTMP config in nginx.conf:

rtmp {
       #access_log logs/rtmp_access.log;
       server {
           listen 1935;
           chunk_size 4096;

           application live {
               live on;
               record off;
           }
       }
}

When the RTMP stops, this is what is logged in the error log (this goes since the fall until we restart nginx):

2024/10/11 10:15:27 [info] 21804#21804: *57 createStream, client: 151.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:15:27 [info] 21804#21804: *57 publish: name='' args='' type=live silent=0, client: 151.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:15:27 [error] 21804#21804: *57 live: already publishing, client: 151.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:15:32 [info] 21804#21804: *57 disconnect, client: 151.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:15:32 [info] 21804#21804: *57 deleteStream, client: 151.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:30:50 [info] 21804#21804: *54 recv() failed (104: Connection reset by peer), client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:30:50 [info] 21804#21804: *54 disconnect, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:30:50 [info] 21804#21804: *54 deleteStream, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:32:34 [info] 21804#21804: *53 recv() failed (110: Connection timed out), client: 151.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:32:34 [info] 21804#21804: *53 disconnect, client: 151.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:32:34 [info] 21804#21804: *53 deleteStream, client: 151.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:41:40 [info] 21804#21804: *58 client connected '88.X.X.X'
2024/10/11 10:41:40 [info] 21804#21804: *58 connect: app='live' args='' flashver='LNX 9,0,124,2' swf_url='' tc_url='rtmp://185.X.X.X:1935/live' page_url='' acodecs=4071 vcodecs=252 object_encoding=0, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:41:40 [info] 21804#21804: *58 createStream, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:41:40 [info] 21804#21804: *58 play: name='' args='' start=-2000 duration=0 reset=0 silent=0, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:43:45 [info] 21804#21804: *58 disconnect, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:43:45 [info] 21804#21804: *58 deleteStream, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:43:45 [info] 21804#21804: *59 client connected '88.X.X.X'
2024/10/11 10:43:45 [info] 21804#21804: *59 connect: app='live' args='' flashver='LNX 9,0,124,2' swf_url='' tc_url='rtmp://185.X.X.X:1935/live' page_url='' acodecs=4071 vcodecs=252 object_encoding=0, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:43:46 [info] 21804#21804: *59 createStream, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:43:46 [info] 21804#21804: *59 play: name='' args='' start=-2000 duration=0 reset=0 silent=0, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:53:35 [notice] 21803#21803: signal 15 (SIGTERM) received from 11550, exiting
2024/10/11 10:53:35 [notice] 21803#21803: signal 15 (SIGTERM) received from 1, exiting
2024/10/11 10:53:35 [notice] 21804#21804: signal 15 (SIGTERM) received from 1, exiting
2024/10/11 10:53:35 [notice] 21804#21804: exiting
2024/10/11 10:53:35 [notice] 21804#21804: exit
2024/10/11 10:53:35 [notice] 21803#21803: signal 17 (SIGCHLD) received from 21804
2024/10/11 10:53:35 [notice] 21803#21803: worker process 21804 exited with code 0
2024/10/11 10:53:35 [notice] 21803#21803: exit
2024/10/11 10:53:35 [notice] 11554#11554: using the "epoll" event method
2024/10/11 10:53:35 [notice] 11554#11554: nginx/1.14.0 (CentOS)
2024/10/11 10:53:35 [notice] 11554#11554: built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
2024/10/11 10:53:35 [notice] 11554#11554: OS: Linux 3.10.0-957.el7.x86_64
2024/10/11 10:53:35 [notice] 11554#11554: getrlimit(RLIMIT_NOFILE): 1024:4096
2024/10/11 10:53:35 [notice] 11557#11557: start worker processes
2024/10/11 10:53:35 [notice] 11557#11557: start worker process 11558
2024/10/11 10:53:47 [info] 11558#11558: *1 client connected '88.X.X.X'
2024/10/11 10:53:47 [info] 11558#11558: *1 connect: app='live' args='' flashver='LNX 9,0,124,2' swf_url='' tc_url='rtmp://185.X.X.X:1935/live' page_url='' acodecs=4071 vcodecs=252 object_encoding=0, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:53:48 [info] 11558#11558: *1 createStream, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 10:53:48 [info] 11558#11558: *1 play: name='' args='' start=-2000 duration=0 reset=0 silent=0, client: 88.X.X.X, server: 0.0.0.0:1935
2024/10/11 11:02:09 [notice] 11557#11557: signal 15 (SIGTERM) received from 11981, exiting
2024/10/11 11:02:09 [notice] 11558#11558: exiting
2024/10/11 11:02:09 [notice] 11558#11558: exit

I had to censor the IP adresses. And nginx service was active in every single moment, never stopped working, it just disconnects the stream.

This is the message that the GoPro shows when the RTMP stream disconnects (remember that the GoPro is still streaming on YouTube with no problem when this happens):
2024-10-11_11-42

We monitor the RTMP live stream with a cronjob running this command, if it doesn't give answer we get an alert, but when it's active it displays some info, we don't know if it can be useful:

[root@streaming ~]# ffprobe -v error -show_format -show_streams rtmp://185.X.X.X/live
[STREAM]
index=0
codec_name=aac
codec_long_name=AAC (Advanced Audio Coding)
profile=LC
codec_type=audio
codec_time_base=1/48000
codec_tag_string=[0][0][0][0]
codec_tag=0x0000
sample_fmt=fltp
sample_rate=48000
channels=2
channel_layout=stereo
bits_per_sample=0
id=N/A
r_frame_rate=0/0
avg_frame_rate=0/0
time_base=1/1000
start_pts=7259968
start_time=7259.968000
duration_ts=N/A
duration=N/A
bit_rate=131072
max_bit_rate=N/A
bits_per_raw_sample=N/A
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
[/STREAM]
[STREAM]
index=1
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=High
codec_type=video
codec_time_base=1001/60000
codec_tag_string=[0][0][0][0]
codec_tag=0x0000
width=1280
height=720
coded_width=1280
coded_height=720
has_b_frames=0
sample_aspect_ratio=1:1
display_aspect_ratio=16:9
pix_fmt=yuvj420p
level=42
color_range=pc
color_space=bt709
color_transfer=bt709
color_primaries=bt709
chroma_location=left
field_order=progressive
timecode=N/A
refs=1
is_avc=true
nal_length_size=4
id=N/A
r_frame_rate=30000/1001
avg_frame_rate=30000/1001
time_base=1/1000
start_pts=7259986
start_time=7259.986000
duration_ts=N/A
duration=N/A
bit_rate=2560000
max_bit_rate=N/A
bits_per_raw_sample=8
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
[/STREAM]
[FORMAT]
filename=rtmp://185.X.X.X/live
nb_streams=2
nb_programs=0
format_name=flv
format_long_name=FLV (Flash Video)
start_time=7259.968000
duration=0.000000
size=N/A
bit_rate=N/A
probe_score=100
TAG:Server=NGINX RTMP (github.com/sergey-dryabzhinsky/nginx-rtmp-module)
TAG:displayWidth=1280
TAG:displayHeight=720
TAG:fps=0
TAG:videokeyframe_frequency=0
TAG:profile=
TAG:level=
[/FORMAT]

If I can provide anything else let me know.
@Jrubensson
Copy link

Jrubensson commented Oct 30, 2024 •
edited
Loading

I have seen similar issues without figuring it out.

Will share here if I come up with something that resolves this, just wanted to let you know.

Currently trying to set max_message 5M; to see if that resolves it.

@cuenta0gow
Copy link
Author

cuenta0gow commented Oct 31, 2024 •
edited
Loading

I have seen similar issues without figuring it out.

Will share here if I come up with something that resolves this, just wanted to let you know.

Currently trying to set max_message 5M; to see if that resolves it.

Hi, first of all thank you for answering!

I already tried setting max_message 5M; for a while, but didn't solve anything.

We found out that SELinux was enabled in the CentOS 7 machine, when we disabled it the problem that made the stream fall every single day at the same time, around 10:20-10:40 and then at 14:00 was gone.

Unfortunately even after disabling that the stream still falls, but only once per day and normally between 00:00AM - 06:00AM. We don't know what causes that. These are the only running services, I think none of these should interfere with live streaming:

[root@streaming ~]# systemctl list-unit-files | grep enabled auditd.service enabled [email protected] enabled chronyd.service enabled dbus-org.freedesktop.NetworkManager.service enabled dbus-org.freedesktop.nm-dispatcher.service enabled [email protected] enabled irqbalance.service enabled kdump.service enabled lvm2-monitor.service enabled microcode.service enabled NetworkManager-dispatcher.service enabled NetworkManager-wait-online.service enabled NetworkManager.service enabled nginx.service enabled postfix.service enabled rhel-autorelabel.service enabled rhel-configure.service enabled rhel-dmesg.service enabled rhel-domainname.service enabled rhel-import-state.service enabled rhel-loadmodules.service enabled rhel-readonly.service enabled rsyslog.service enabled sshd.service enabled systemd-readahead-collect.service enabled systemd-readahead-drop.service enabled systemd-readahead-replay.service enabled tuned.service enabled vgauthd.service enabled vmtoolsd.service enabled dm-event.socket enabled lvm2-lvmetad.socket enabled lvm2-lvmpolld.socket enabled default.target enabled multi-user.target enabled remote-fs.target enabled runlevel2.target enabled runlevel3.target enabled runlevel4.target enabled unbound-anchor.timer enabled

EDIT: I'm adding a screenshot since the pasted code doesn't look very good:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Jrubensson @cuenta0gow