Dual video playback with HDMI audio output get stuck when one of them is paused

[dividuum]

Hi.

I encountered a problem when having multiple OMX pipelines setup like this:

video_decoder -> video_scheduler -> video_render
                        |
                      clock
                        |
                        +---------> audio_render  

I start playback on each pipeline by feeding video data and decoded audio data into the buffers of video_decoder and audio_render. This works fine and I can play two videos at the same time without problems.

The problem is when both audio_render components are configured to send audio to HDMI using OMX_IndexConfigBrcmAudioDestination. Once I do that and stop one of the clocks by setting its xScale value to 0, the other video gets stuck after a few moments. It seems that it can't request another audio input buffer from the audio_render component and the complete pipeline stalls. Once I restart the clock of the other video, both videos continue.

First I thought I might have some problem in my code, but I was able to reproduce the problem with omxplayer: You need a longer video with sounds. I used the I am legend trailer from here: http://www.dvdloc8.com/clip.php?movieid=13599&clipid=4

Then in two terminals run these commands:

 omxplayer -o hdmi --win 0,540,1920,1080 trailer.mp4

and

 omxplayer -o hdmi --win 0,0,1920,540 trailer.mp4

Now pause one of them by pressing space and wait for approximately 10 seconds: The other one will get stuck. It takes a bit longer to reproduce with omxplayer, since it seems to allocate more audio buffers.

When switching any or both of the audio targets to local it works as expected, so it seems HDMI related.

I tested on Raspbian wheezy

Linux raspberrypi 4.1.17-v7+ #843 SMP Mon Feb 15 23:35:33 GMT 2016 armv7l GNU/Linux

Feb 15 2016 23:38:07 
Copyright (c) 2012 Broadcom
version 263c9ef2f99450604d945968d75c013abf1fc841 (clean) (release)

omxplayer - Commandline multimedia player for the Raspberry Pi
    Build date: Sat, 12 Sep 2015 00:49:19 +0000
    Version   : d99bd86 [master]
    Repository: https://github.com/popcornmix/omxplayer.git

[popcornmix]

See: popcornmix/omxplayer#317
no solution currently.

[dividuum]

Thanks for referencing that other issue. I only searched through firmware issues.

As mentioned by another poster in the referenced issue: It seems to work when using the jack audio output. Is that coincidental and might break in the future or are the two audio implementations (hdmi vs local) different in the way they implement pausing? Can I rely on local output working as intended?

How much do I have to pay you to fix the problem with HDMI audio? :-)

[popcornmix]

A quick look at the analogue audio driver shows analogue behaves like hdmi when force_pwm_open=0, but with force_pwm_open=1 (the default) it implements pause differently.

I'm not currently sure if hdmi could use this scheme for pausing. I've added it to my list of things to investigate, but as it's a pretty niche use case, this isn't very high priority.

[dividuum]

Thanks for investigating! It would be really awesome if pausing would work with HDMI as well, as that limitation seems arbitrary and seamlessly playing videos & audio over HDMI is a useful feature.

Meanwhile: Is there any obvious workaround I'm missing? I cargo culted a bit and OMX_StatePause'ing the audio_renderer before stopping the clock seems to work. At least the other video continues to play. But I can't tell if audio works, since I don't have an HDMI display with audio output right now (which is also the reason why I didn't notice this whole problem earlier).

If I understood your suggested solution in the reference issue correctly, I'd have to go for a purely arm side clock that replaces the OMX clock component, so I can schedule video and audio (which I then software-mix?) myself. That seems like a major change which I'd love to avoid if possible. Any other hints?

[dividuum]

Any chance this issue might get some love? I'm still stuck with either implementing an ugly workaround which will probably break in the future or having partially broken HDMI audio support in my program.

My workaround right now is to single step the decoder for 1 frame. When the frame reaches the renderer I pause the audio component, then stop the clock. This leads to a small bit of audio (probably 1/60s for 60fps videos?) playing before I can stop the clock which I could mute so it's not noticeable. But of course this results in a single dropped "audio frame" which is noticeable when trying to do seamless video playback. As I said: ugly. Any hints on improving this are more than welcome.

[incaseoftrouble]

Bump. I'm also having this problem and would love to have it fixed - any chance this will happen?

[Tillman-Z]

Bump. I second that. Would love to see this issue go!

[lunanigra]

Please fix! HDMI is getting more and more important... (S-Video already got removed)
See iPhone 7, no future for jack audio output.

[dividuum]

Is there any chance this might get fixed at some point? Frame-exact video playback is difficult to do without that, as I have to preload a video file, stop playback and then start at the exact moment in time. Due to this problem this doesn't work with HDMI enabled unless it's the only video loaded/playing. Which is something I can't guarantee due to the dynamic nature of my tool. Is there anything I can do to get this fixed?

[paolosabatino]

Me too would like this problem fixed

[dividuum]

Any chance this might get fixed?

[JamesH65]

I'm taking a look. The code is fairly impenetrable, and uses DMA to make things even more complicated, but I am starting to get a vague idea on how it works. Got something to try today, which may or may not have some effect. If it doesn't, not sure where to go from there!

[dividuum]

Thanks a lot for taking a look. I suspected that this isn't an easy problem to solve and probably requires quite a bit of reorganizing. Happy to test any new version!

[JamesH65]

Hmm, my attempted fix does seem to work. Which is unexpected. I'd like someone with more experience of the issue to test the firmware, I've just tested with 2 OMXplayer instances. I'm just building a release, I'll upload to dropbox or similar sometime today.

[dividuum]

That's some existing news! Thanks again for taking the time to look into this.

[JamesH65]

OK, can you try this firmware.

https://www.dropbox.com/s/6ocsedtam6hs10u/hdmi_audo_pause_firmware.zip?dl=0

Note, I have included start.elf and start_x.elf, as I don't know which you use. You may also need to appropriate fixup.bat files as well. Good luck!

[dividuum]

I'll have a look at the new firmware later today. Thanks!

[dividuum]

This actually works for one of my previously broken test case. Awesome! What a nice present!

What are the steps going forward? Is this stable already or more like a proof of concept? Out of curiosity: What did you change? Decouple the audio mixing(?) from the 'audio' sinks/clocks?

[JamesH65]

That's good news! I'd like as much testing as possible, especially in the area of syncing audio to video when rapid pausing.

I believe the fault is to do with reference counting the number of pauses, compared with the number of instances. Only when the number of pauses equals the number of instances should it actually stop the HDMI output completely (or replace with silence). It was stopping output on the first pause instead. So an easy fix compared with what I originally thought was the problem , that of the DMA being stopped or similar.

[dividuum]

Oh. That sounds easier than I expected. Nice catch. I can definitely do some more testing.

[dividuum]

So far in all my experiments everything works pretty well. Any idea on when this fix might be included in an official release?

[dividuum]

It seems that this fix still didn't end up in

Mar 14 2018 16:50:42 
Copyright (c) 2012 Broadcom
version 5afe871e8e58f66e0a39531895d09ed692893267 (clean) (release)

Any chance it might be included in a future release?

[JamesH65]

I thought it was already included! @popcornmix @pelwell Did it get lost, or did I forget to PR it?

[6by9]

@JamesH65 Look at gitlab merge request 102!

[JamesH65]

Ah Ha. Should now be going through the merge process. Not sure when it will appear in public, but its on its way. Probably less than a week to hit rpi-update.

[dividuum]

Thanks @JamesH65 and @6by9!

[dividuum]

I've run into another very similar problem and I would suspect that it's the same underlying problem. Try the following:

terminal 1 $ omxplayer video-with-audio.mp4 -o local --win '0 0 320 200'
(after one second)
terminal 2 $ omxplayer video-with-audio.mp4 -o local --win '400 0 720 200'
(after one second)
<press space in terminal 2> ➜ the video in terminal 2 pauses as expected
(after a few seconds)
<press space in terminal 2> ➜ the video in terminal 2 unpauses as expected
                              but the video in terminal 1 pauses!?

This only happens when local is selected as audio target. When using hdmi it works as expected. Any idea what's going on? I'm testing on the latest apt-getable Stretch kernel/firmware:

Linux raspberrypi 4.14.34-v7+ #1110 SMP Mon Apr 16 15:18:51 BST 2018 armv7l GNU/Linux

Apr 16 2018 18:21:52 
Copyright (c) 2012 Broadcom
version af8084725947aa2c7314172068f79dad9be1c8b4 (clean) (release)

[JamesH65]

Sorry about delay - missed the follow up question.

I can replicate this, and I suspect you are right that it is similar. The local driver doesn't have any reference counting either. I'll drop some in, and see what happens.

[JamesH65]

Hmm, similar fix as the HDMI doesn't fix the issue, but not sure why...will keep looking.

[JamesH65]

OK, found it. Have asked for a fix to be merged to our firmware tree. Still awaiting review, but hopefully in rpi-update in the next week or so.

[JamesH65]

Merge has now been done, so will be in rpi-update in the next few days. Would apreciate some testing on this, but in my checks all seemed well. If happy, please close the issue.

[popcornmix]

rpi-update firmware contains @JamesH65's fix.

[dividuum]

I've just tested this with both analog and HDMI output and both cases now work as expected. Thanks for fixing. Bug report can be considered closed.

[JamesH65]

Cool, thanks for testing.