docker version orkaudio maybe memory leak #106
Comments
|
only 10 cps, the memory useage keep grow always, and never reduce
|
|
|
|
Your second graph shows the the graph flattens around 2GB. Any chance that you have kept it running for longer duration and observed the usage? |
|
@kingster thanks for reply. the second graph show on 50 cps stress test. the graph flattens around 2GB, because i stop the stress test. the fire graph show on 10 cps stress test |
|
this is my config.xml |
|
the orkaudio will be killed because the Out Of Memory limit of the docker service. so it can not be runing long time |
|
i think the speed of memory grow is not normal. |
|
i use sipp stress test |
|
this is start log |
|
Can you share your sipp stress scripts, so that I can reproduce this issue? I am running a slightly older version on production (v0.2.5) and haven't observed any memory leak, so it's possible that the memory leak got introduced in the recent merges from upstream. |
sipp uas |
|
Thanks, I will try to reproduce the issue in our setup. |
|
docket stats show orkaudio use 724Mib but when get into the contatiner and use top, the orkaudio only use 233MB when out of the container, htop show orkaudio only use 227MB
in the container, the top show orkaudio memory usage will be stable after a while. but docker stats show orkaudio's MEM USAGE will grow always, when it hit the memory limit, docker will restart orkaudio. |
|
the metrics container_memory_working_set_bytes from cadvisor is keep grow
|
|
I think this is could be an explainable behaviour as you confirmed the memory usage of orkaudio is stable from the top output. Given that the recorder is continuously writing recorded files, the file cache would also be reported as memory used. Have a look at this issue moby/moby#40415 which exactly talks about this behaviour. |
|
i set a cron job in the orkaudio container the memory usage will be release erery minute. but as long as i keep stress test it, the memory usage keep grow very slowly, about 1 mb grow every minite, but i think it is ok.
but what i realy don't understand why after the stress test, the memory usage keep a stable level (300MB) and never go down. The behavior I expect is that after the stress test is over, the memory usage return to a lower level, not 300MB
|
|
@kingster do you use production v0.2.5 is a docker container? or just a install version? i also test docker orkaudio:0.2.5, the memory is keep grow very slowly too |
wangduanduan
changed the title
|
my stress test is 500 concurrent call, every second make 25 calls, every call duration is 20 seconds, media type is g711, what memory it should use? |
|
The original docs mentions about In the real world, oreka is much lot more CPU intensive unless you have unlimited disk and continue to record on In our production environment, we run the native version on bare metals, given the CPU utilisation it has (we transcode to compressed
I would suggest you start with the original recommendation of 4GB, and then tweak based on your utilisation. |
|
In my experience (we use a very customized version of OrkAudio in production, more than 500 servers), these specs are way high. Out of all the threads used in OrkAudio, only the "batch" threads use more of the CPU/IO when they need to transcode from MCF to whatever format you need (stereo WAV) in our case. That said, we will also investigate the memory leak in our version. Usually, we crank the number of batch threads in sites where the traffic is high, more than 400 calls, the magic number is 4 threads in a server with 4 cores and 16 Gb of RAM. Right now, I just check this server and the RAM usage is below 1.3 Gb, the peaks in CPU usage are due to batch threads. |
|
Some metrics from one of our production recorder (Post Opus Memory fixes) , our memory utilisation looks sort of constant, doesn't increase and has very slight decrease . Call Rate: ~400calls/min, avg call duration ~1min, i.e concurrent ~400 calls, being transcoded to opus codec. After about few hours.., Following are some of metrics of the bare-metal.
Memory Utilisation over 24hrs |
|
i have this leakage as well it seem it's from AudioTape class which holds all the audio chunck |
docker voiceip/orkaudio:master
only 10 cps, every call talk 10 seconds, the memory useage keep grow always, and never reduce
The text was updated successfully, but these errors were encountered: