[HLS] Player is not able to pick up an audio track from the original video

[gjanblaszczyk]

Have you read the FAQ and checked for duplicate open issues?
Yes

If the problem is related to FairPlay, have you read the tutorial?
Yes

What version of Shaka Player are you using?
4.5.0

Can you reproduce the issue with our latest release version?
Yes

Can you reproduce the issue with the latest code from main?
Yes

Are you using the demo app or your own custom app?
Demo app

If custom app, can you reproduce the issue using our demo app?
Yes

What browser and OS are you using?
Chrome 118.0.5993.117

What are the manifest and license server URIs?
HLS playlist from the official Apple website. For example this manifest:
https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8
(from https://developer.apple.com/streaming/examples/basic-stream-osx-ios5.html)

What configuration are you using? What is the output of player.getConfiguration()?
default one.

What did you do?
Run the above HLS playlist on the demo website.

What did you expect to happen?
The original audio track from the main video is visible in the player language menu.
According to HLS manifest:
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="eng",NAME="BipBop Audio 1",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="eng",NAME="BipBop Audio 2",AUTOSELECT=NO,DEFAULT=NO,URI="alternate_audio_aac/prog_index.m3u8"
We should have two audio tracks available on the menu but the player shows only one. BipBop Audio 2

What actually happened?
We should have two audio tracks available on the menu but the player shows only one. BipBop Audio 2

After some investigation, it looks like the player ignores audio with the name BipBop Audio 1 because it doesn't contain a URI
which I guess is wrong because this audio track is built into the video stream so it should be available inside the player. Any thoughts?

[avelad]

We currently do not support muxed and non-muxed content in the same stream. That's why your content doesn't work.

Do you want to contribute (PR) to having this functionality?

Thanks!

[gjanblaszczyk]

Hi @avelad,
I am not very familiar with this project.
I wonder, how much work is needed to implement this functionality in the player.
Are you able to give me some raw guesstimate? please

Thanks!

[nic6200]

Hi @avelad.

I was trying to see if I can help here (new to this repo). According to the HLS spec:

   If the media TYPE is AUDIO and the URI attribute is
   missing, clients MUST assume that the audio data for this Rendition
   is present in every video Rendition specified by the EXT-X-STREAM-INF
   tag.

Since we're filtering out any audio rendition without a URI here, we end up only playing the one with a URI.

Please correct me, if I misunderstood anything. Thanks!

[joeyparrish]

The only way to support a muxed video+audio stream and a separate unmuxed audio stream would be to demux the video+audio stream internally, so that we could choose which audio to send to MediaSource. Without that infrastructure, we can't choose between them equally.

I don't know how much work that would be, but since we now have our own transmuxer, it should be more approachable than it was when we were stuck with a third-party transmuxer. (It might still be a lot of work, though. Not my area of expertise any more.)

[avelad]

@joeyparrish
Best example of current implementation: https://01.cdn.vod.farm/testing/b2ac102ac3f1434504bf5e43d957bc8b/master.m3u8 --> lower qualities muxed in TS (H264+AAC), 1080p muxed in MP4 (AV1+AAC)
For TS muxed content we separate Audio and Video in MediaSource and the transmuxer only generates audio or video (see https://github.com/shaka-project/shaka-player/blob/main/lib/transmuxer/ts_transmuxer.js# L204 and https://github.com/shaka-project/shaka-player/blob/main/lib/media/media_source_engine.js#L494)
When we switch between TS and MP4, we restart the media source, see https://github.com/shaka-project/shaka-player/blob/main/lib/media/media_source_engine.js#L1933
In short, we support splitting TS content into audio and video(two mp4 s), we support codec change, we support media source reset
There is a quick solution, although not optimal...
It would be to create an audio stream that has the media playlist of the video. But we would request the segment, the playlist twice, but it would work without any changes, except forcing to use transmuxer in these cases.

[nic6200]

The only way to support a muxed video+audio stream and a separate unmuxed audio stream would be to demux the video+audio stream internally, so that we could choose which audio to send to MediaSource.

@joeyparrish Please correct me if I am wrong - When I play the above URL, the current player plays BipBop Audio 2 (separate unmuxed audio) along with one of the muxed audio+video streams.

In this case, the audio stream in the muxed stream is BipBop Audio 2 (since this stream doesn't have a URL).

[joeyparrish]

I haven't debugged this, and my memory might be out of date since @avelad is so prolific... but I believe that we are sending the audio+video stream to MSE where it may be treated as video-only inside the browser based on how we configured it. This would make the audio-only stream the only audio stream available.

[avelad]

Live stream with this case: https://ztnr.rtve.es/ztnr/1694255.m3u8