-
Notifications
You must be signed in to change notification settings - Fork 314
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ffmpeg eats all the memory and crash within a minute - recording or streaming #269
Comments
We seem to have found similar or the same issue with Jitsi/Jibri on Debian 10, so far as regards the memory usage of That said, I've not yet tried running without |
Seems to be it ! Also, a third person reported the same on the forum
In the general case, I think you need to modify the jibri source code. In my case, just for testing purpose I did the following quick hack: rename /usr/bin/ffmpeg to /usr/bin/ffmpeg-original and then create the following /usr/bin/ffmpeg script
So it's not a solution by any means, because jibri can't even stop ffmpeg at the end of a session anymore, just a quick check to confirm sound is the issue. |
|
I have the exact same issue. |
Here the same. Can anybody confirm this? |
I can confirm that, when you disable third party requests, the memory usage seems to be stable. |
Ok, but that only seems to have been the beginning. After 26-27min, the memory went through the roof from 1.5GB to 8GB and Swap. The same stream, no interaction, hardly any sound. |
We have have tried setting |
Thanks for the suggestion. We have tried a similar script (in |
Ok, here's another try. I've changed the ffmpeg version which is come from Ubuntu 18.04 Repo in version
up to...
with
Memory is the mostly time 1.7GB. Sometimes it goes up to 3GB for no apparent reason, but quickly returns to 1.7GB. Edit: CPU load 5-10% better. |
After 30 min without reason out of memory and swap. Incredible. |
Ok, I'm breaking off now. Video recording is great feature, but not particularly important for me at the moment. 50min stream no problem. After restarting the server, it no longer works properly. I set it up to a working videochat without recording stream. Bye Guys and good luck. |
For us, it's specially important. |
I no longer thought about compiling it myself. Then you should also be able to set your own parameters. Definitely exciting. I'll keep my fingers crossed. |
Hi, i have the same problem, after a moment memory go full and crash ffmpeg. |
I finally compiled the latest version of ffmpeg and now everything is fine. |
It's good to hear you found a way to resolve this; although we haven't been able to replicate your fix so far. Which version of ffmpeg did you compile? Is this still on Debian 10? We too are running Debian 10 ( Firstly we tried release 4.2.2, which seemed to result in the same behaviour as before ‒ the Second we tried the git master build dated As an aside, before your last comment I'd thought the ffmpeg version was unlikely to be the cause given that we seem to have a variety of versions floating around including those in the repos of Ubuntu 16.04, Ubuntu 18.04 and Debian 10 as well as some from third-party repos. In addition the install guide at https://github.com/jitsi/jibri refers to adding a repo to get a newer ffmpeg version when using Ubuntu 14.04 (which is EOL since last year) but that only provides ffmpeg version 3.4.0 which is older than the versions from the repos of Ubuntu 18.04 and Debian 10. |
I compiled the latest ffmpeg available as of yesterday, following the guide at: https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu We are at Debian Buster, and I used the following settings:
The resulting ffmpeg version is N-97465-g4e81324 I just tried to record a session for an entire hour, and it worked very stable. |
I tested now streaming to youtube. |
Has this been fixed? I was about to install jibri but after reading this I'm a bit weary of how much memory is actually needed. Anyone have a working setup? If so, what hardware specs? Thanks |
To answer my own question in order to help others, no it hasn't been fixed. I recorded a 1.5 hour long conference yesterday and I had a bunch of errors like the ones described here. I'll see if I can dig into the logs when I have some time and post them here. This is very disappointing as there doesn't seem to be any solution available. |
I having this same problem... |
We have the same issue on Kubernetes (GKE), but strangely enough, not Docker. The Kubernetes Jibri deployment managed to OOM a node with 6.5 GB RAM with a single livestream. The Docker deployment is running at 1.15 GB. Both are using exactly the same |
We may have found a solution/workaround for this, although it's a surprising one. In a nutshell, it's using a Java 8 JRE instead of a Java 11 JRE. This despite the high memory usage we've seen before being by either FFmpeg or Xorg, neither of which are Java processes. The Jibri service truns in a JRE, though. We came across this when looking for differences between the Jibri Docker image and our 'native' (i.e. not container based) installation on Debian 10. The Jibri docker image is based on Debian 9 where Another clue was Woodworker_Life's How-to to setup integrated Jitsi and Jibri for dummies, my comprehensive tutorial for the beginner thread on the Jitsi Community Forum. There Woodworker_Life appears to literally say that Jibri won't work with a Java 11 JRE, although not specifically how/why. There is no OpenJDK 8 JRE in the Debian 10 repos and so, like in Woodworker_Life's thread, we used the OpenJDK 8 JRE package for Debian from the AdoptOpenJDK project. So far we've several long attempts at recording and streaming (to YouTube), albeit most only with 2 participants. For at least one we've run for over an hour. In no case of those cases have we had memory issues or crashes. Memory seems to hold steady sometimes as low as 6xx MB, but always less than 2 GB so far. That said, for real-world use where we'll have more participants (albeit < 10) we won't skimp on memory allocated for the VM used (we'll have 6 GB). We'll shortly be building and working with the server for 'real world' use, so we'll update if we again encounter issues ‒ but, fingers crossed, it's looking good! |
My case: Debian 10 with Java 8. Test: youtube live streaming. Machines from GCE. 2 vCPUs and 6GB RAM -> eventual crash My theory is that a low number of CPUs makes ffmpeg not to be able to process data as fast as it should. |
We had a very frustrating experience this week with Jibri. We had tested it with our latest specifications (Debian 10, Jre 8, 4 vCPUs and 12GB RAM) for several recordings that went well, so we decided to go on and scheduled a very important meeting to stream it to YouTube. The meeting was for 27 participants, 14 of them visible (LastN=14) and was going to be very long (about 5 hours). |
Have you searched around for anything on the ffmpeg side for this? Seems like such an odd bug. I don't think it can be a fundamental parameter issue, as we're not seeing this consistently, but maybe a combination of some setting we pass to ffmpeg and some network condition? |
we have the same issue, our configuration is Ubuntu Server 16.04 and ffmpeg 2.8.15-0ubuntu0.16 and jibri 8.0.30,any solution? By the way in other enviroment with the same ffmpeg and ubuntu versions this does not happen , the difference is the Jibri Version in this enviroment is 7.2.71-1 |
We've found the same issue on a local Kubernetes cluster. It's an Ubuntu 18.04 based cluster, with Jibri compiled 6 weeks ago from testing release. Jibri was recording directly into a NFS folder. In our case, the Kernel wasn't flushing cache memory quickly enough so OOM_Killer got triggered. We fixed that situation by moving the recordings folder to a local folder of the node and then moving the recording to NFS. This way, the Kernel is behaving properly and never fills the machine, despite RAM consumption is huge, anyway (Cache, not RSS). What's weird is that the exact same Jibri deployment works fine on other Kubernetes, so, maybe it's related to something else (base Kernel, CPU power....). Still investigating. In any case, maybe you can try to increase cache pressure in your kernels to avoid filling up your memory: vm.vfs_cache_pressure=150 or 200 |
@kpeiruza I increased cache pressure in my Ubuntu kernels and and it didn't work. |
Thanks a lot @sblotus you helped me to nail it down. So I just finished my tests. Basically, last time, it worked, because I fiddled with ffmpeg scripts, thanks to @emrahcom and I did put something like that:
And the previous version, it was also sending 720p instead of 1080p. But basically the "memory leak" comes from our ffmpeg not being fast enough.. So the only thing you have to watch is:
If it looks like this:
It means it is not fast enough, and the queue is accumulating in memory. Then it crashes. So either you fiddle with the ffmpeg args (definition, preset..) or you need better hardware. I also checked that with a hetzner VM non dedicated cpu, it doesn't work, whereas with a dedicated modern cpu, it does. I think we can close this issue, as we just need to have a faster cpu to be able to encode faster, or less quality for ffmpeg to be able to be up to date. Thanks again @sblotus and @emrahcom for your help :) |
not sure
|
This does not work for me - lasts 3 minutes and 38 seconds (ran three times) & consistently fails. 2021-05-27 18:34:14.556 INFO: [59] ffmpeg.log() ffmpeg version 4.3.2-0york0 I have installed Ubuntu 18.04, latest JIBRI/ffmpeg chrome 90 etc on an old Gen 4 Intel i3 with 4GB RAM & it just works, 24x7 - sample log files - 4 cpus at 50 to 60% constantly less than 1GB RAM used 2021-05-27 18:41:35.161 INFO: [55] ffmpeg.log() frame=732090 fps= 30 q=24.0 size= 7414826kB time=06:46:42.98 bitrate=2489.1kbits/s speed= 1x |
In both cases your ffmpeg speed is < 1x so then queuing in mem, looks like a leak, but it is just normal, you need to tune your VM and ffmpeg to have a speed of 1x :) |
@pierreozoux any particular settings for VM ? We have tried many, some work for an hour or two but none so far that provide 24x7 reliability. |
@pierreozoux in hetzner, using a VM with dedicated CPU worked, whereas non dedicated didn't. |
On GCP/GKE, we've had much better luck with AMD Epyc machines (N2D) than standard ones (N1 - Intel up to Skylake). We haven't done extensive testing, but with 2 cores and 4 GB RAM, N2D nodes could run Jibri for over half an hour while N1 nodes with the same or even better specs overloaded and crashed within minutes. If your CPU is not fast enough, frames will start buffering in RAM - it's as simple as that, as far I understood it. Shared vCPUs are a no-go for any serious workload on any provider, this should be obvious. Their performance is extremely inconsistent. |
@saghul I think we can close this discussion and continue in the forum if needed. |
@pierreozoux @saghul further testing reveals -
All tests repeated / repeatable. So other than bare-metal i3 4th Gen this is quite an issue. |
Same.
|
You certainly cannot follow the principal of fire up another JIBRI VM when the others are busy - unless its a Wake On LAN command to fire up another i3 old desktop pulled in for use ..... Bare metal is the only 100% stable option. |
No, it was simple test with 1 conf and 1 VM. |
Sorry was not referring to you directly but metaphorically "no one" can follow the principal as it's just too unreliable. |
Do you know that the default resolution changed? So if your box can't stream 1080p, try 720p. I just did the test with a box that can't stream 1080p, it can 720p :) |
Yeah fully aware. It's not about "a box" it's about VMs in general not being reliable .... full stop on ESXi/AWS/XEN/Hyper-V etc. My base piece of bare-metal for testing is a "lowly" Intel(R) Core(TM) i3-4130 CPU @ 3.40GHz with 4GB RAM. This can handle JIBRI all day long, 24x7 - longest stream is 48 hours (max 1.4GB RAM used), including to an nginx RTMP server running on the same box. Full resolution 1920x1080. So if you have a dual CPU Xeon (32 cores, 192GB RAM) - you'd expect at a minimum 3 or 4 JIBRI to run quite happily (in theory a lot more). So, it appears that "recommended" are 4 cores / 8 GB RAM on Hypervisors per VM. What @trashographer and I are saying is that with just one JIBRI with pretty much zero load it just dies very quickly. Even if you bump to 8 cores & guarantee 50% CPU or greater it dies still fairly quickly. If bare-metal works on such lowly specifications then why is it we should "mess around" with lowering resolution as I have done that and can still get failures within one hour. |
In my case i install jibri docker instances. I have same issue. I solve it by installing older version of docker instance. Actually 8 month ago version |
wow, thanks |
Could you tell me what tag to use? for jibri in docker |
Now it still works for you? Could you share your Docker configuration in a repository, please, it would be helpful. Thanks. |
I use this image |
jitsi/jibri:stable-5076 |
The latest version now defaults to 720p, I'd encourage you to try that. Soon enough the Chrome version on that old image will be too old to run. |
@McL1V3 @carellano @DonxDaniyar Kindly can you confirm if this worked for you? Thanks. |
That release is almost a year old, i'd suggest you test with the latest image. |
I have a jibri image with docker customized with chrome 101. Recording at 720p, very little happens to me but as I have 5 jibris working, in a last update if one crashes in a live recording the jibri ends and uses another jibri, like this successively I have practically no video loss and it has been working without problems for several months |
1080x720 working |
Description
On jitsi when I start a recording or a streaming session, in less than a minute the recording/stream will stop and my whole server become slow and unresponsive.
With top, I could pin the culprit: ffmpeg. It eats away all the memory very quickly. In less than a minute my 8GB are filled.
You can find attached the log of jibri when I tried a streaming session. Nothing stands out to me. I stopped the streaming after 15 seconds and ffmpeg was already at 40% memory.
Also if I stop completely prosody, jicofo, jvb and jibri and if I log as a jibri user and starts ffmpeg by myself, using the command I found in log.0.txt, I get the same issue, the CPU shoot to 150% and the memory keeps growing. I have to kill ffmpeg before it saturates the memory.
ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:bsnoop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -maxrate 2976k -bufsize 5952k -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f flv rtmp://a.rtmp.youtube.com/live2/aaa
If I remove every parameters related to sound in this ffmpeg command line, so removing -f alsa -thread_queue_size 4096 -i plug:cloop -acodec aac, then the memory saturation issue goes away. Memory usage is stable. So It clearly seems to be related to the sound. How can I debug this kind of issue ?
Possible Solution
Steps to reproduce
Environment details
Ubuntu 16, followed the instruction on github
browser.0.txt
log.0.txt
ffmpeg.0.txt
asoundrc.txt
The text was updated successfully, but these errors were encountered: