Pi 4 - bcm2835-codec - Driver leaks buffers because frame completion might be too aggressive #3325
Comments
|
We see the same error message when trying to seek with a fairly simple pipeline: The decoder v4l2h264dec seems to be the issue. Please also see this thread on gstreamer-devel: Seeking is currently not possible on the Raspberry Pi 4. Would be really cool to get some feedback here. |
|
I recently had to re-apply this patch on a couple of devices I had upgraded. For anyone finding this, I'm building an image via pi-gen and here's what I'm doing (you can probably also use it a on a running device with Raspbian): firmware=$(zgrep "firmware as of" \
"/usr/share/doc/raspberrypi-kernel/changelog.Debian.gz" | \
head -n1 | sed -n 's|.* \([^ ]*\)$|\1|p')
kernel="$(curl -s -L "https://github.com/raspberrypi/firmware/raw/$firmware/extra/git_hash")"
# We only care about the Pi4 here
uname="$(curl -s -L "https://github.com/raspberrypi/firmware/raw/$firmware/extra/uname_string7l")"
KVER="$(echo ${uname} | grep -Po '\b(Linux version )\K(?<price>[^\ ]+)' | cat)"
mkdir -p /tmp/linux
cd /tmp/linux
git init
git remote add origin https://github.com/raspberrypi/linux.git
git fetch --depth 1 origin ${kernel}
git reset --hard FETCH_HEAD
KERNEL=kernel7l
ARCH=arm
make bcmrpi_defconfig prepare
sed -i 's/wait_for_completion_timeout(&ctx->frame_cmplt, HZ)/wait_for_completion_timeout(\&ctx->frame_cmplt, 10 * HZ)/g' drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c
cd drivers/staging/vc04_services/bcm2835-codec/
make -C /tmp/linux M=$(pwd) modules
cp bcm2835-codec.ko /lib/modules/${KVER}/kernel/drivers/staging/vc04_services/bcm2835-codec/bcm2835-codec.ko
mkdir -p /lib/modules/${KVER}/extra
cp bcm2835-codec.ko /lib/modules/${KVER}/extra/bcm2835-codec.ko |
|
@nemosupremo Wow! You're my hero! This fix works! We will add it to our pi-gen scripts as well. Thanks a lot for sharing! 👍 |
|
OK, I'll bump the timeout up a bit. It shouldn't take that long to flush though, so something is a bit odd. GStreamer 1.16 includes a couple of fixes to avoid allocating a stupid number of buffers (basically filling memory) after a seek. |
hmm, right, sorry about that, we didn't have codec with CREATE_BUFS support when 1.14 came out, and never notice that one. |
No issue at all. AFAIK What GStreamer was doing was valid within the V4L2 API spec, just unexpected. I expected the number of buffers specified in queue_setup to be the number passed back in start_streaming, but it wasn't in this case. |
|
I have a similar (maybe even identical) problem with latest GStreamer 1.19 and latest Kernel 5.10.17 in bug #4194 . |
|
Hi, I'm still able to reproduce the issue with the current 2-second timeout, and this looks like a firmware issue IMHO.
Also tried with timeout as 10, 20, and even 30 seconds, and they don't really help with the issue. Tested with some newer version firmware, the issue can still be reproduced:
|
|
There is no leak when that WARN triggers. Videobuf2 cleans up all the buffers that are in the wrong state on behalf of the driver. |
|
Important to mention is the Raspberry Pi version. I've been looking into this issue for a while and found out some people have resolved this either by upgrading firmware/kernel or by increasing the timeout in the codecs driver or increasing the overvoltage settings. And some other people have never resolved this issue in any of the mentioned ways. The difference between them is that the problem seems to be resolvable on Rpi 4 but not on Rpi3+ or Compute Module 3+. I am sure this is why this bug persisted opened for two years. It could even be a hardware issue, but when I use userland, all is fine. So I guess it is only some undocumented quirk of the BRCM chip. |
|
Is this the same error? https://gist.github.com/ibash/bdbdfac2a1b46a406f681460e122aadc On a pi4:
|
|
#4765 should deal with the leak, but not merged yet. |
|
@ibash It's the same error being reported, but for different reasons. |
|
Hi @ibash and @6by9 , thank's for reporting. My understanding is that it is the same error. It just doesn't matter where the stream comes from, whether it was from the camera or from the file. From hw codecs perspective it is the same workload. I want to try the patch mentioned above. It seems interesting. |
Describe the bug
I have a Pi 4 running WPE Webkit v0.22.5 on Raspbian Buster. I am playing videos, in a loop, on the HTML5 browser. WPE Webkit uses gstreamer internally which uses the v4l2 stack to play hardware accelerated video. Every time the video stops running, the driver leaks 2 buffers with the message
bcm2835-codec: bcm2835_codec_stop_streaming: Timeout waiting for buffers to be returned - 178 outstanding. Eventually when there are ~6000 buffers outstanding, the device becomes unstable and refuses to play anymore video. Kernel timeouts also start to happen.To reproduce
There is no straightforward way to reproduce the issue without building WPEWebkit / Cog / Weston, which takes hours. I tried to reproduce the issue with
gst-play-1.0, but I wasn't able to reproduce it.Expected behaviour
The buffers are not leaked.
Actual behaviour
Buffers are leaked
System
cat /etc/rpi-issue)?vcgencmd version)?uname -a)?Linux raspberry 4.19.81-v7l+ #1276 SMP Fri Nov 1 14:23:48 GMT 2019 armv7l GNU/LinuxLogs
Additional context
I was able to alleviate issue by increasing the
wait_for_completion_timeoutto10 * HZ. I've been running the system for several days now and I no longer see those messages.The text was updated successfully, but these errors were encountered: