Playback fails for stream with different video and audio offsets

[evanfarina]

Description

The stream in question has a video PTS of 25.385 and an audio PTS of 3.392. In Chrome (test URL, the video will spend three seconds trying to play before it ultimately stalls out.

Sources

https://livectorprodmedia11-usw22.licdn.com/abc3683d-fa9a-4594-8728-456279f6bf4f/L565cdf276e7902c000-livemanifest.ism/manifest(format=m3u8-aapl)

Steps to reproduce

1. Visit the http-streaming demo page
2. Enter the above source into the box for the media's source
3. Click the "load" button and then attempt to play the stream
4. Allow the stream to play for ~3-4 seconds. Notice that the stream will stall and the loading spinner will appear. The stream will not recover from this point on

Results

Expected

I expect the stream to play without stalling out.

Error output

No error output

Additional Information

@gkatsev helped me track down the logic where the timestamp offset is looked up. We found that this stream does not specify an offset. Therefore, it is my expectation that the video and audio be properly muxed in this case. In other words, I do not think this is an encoding issue but rather an issue with the player's muxing logic.

Also helpful, here is the output from ffprobe

First video fragment:
pkt_pts_time=25.385000
pkt_dts_time=25.452000

First audio fragment:
pkt_pts_time=3.392000
pkt_dts_time=3.392000

videojs-http-streaming version

2.2.3

videojs version

7.10.1

Browsers

what browsers are affected? please include browser and version for each

• Chrome 86.0.4240.111

Platforms

what platforms are affected? please include operating system and version or device and version for each

• Mac

Other Plugins

are any other videojs plugins being used on the page? If so, please list them with version below.

• No plugins are being used

[welcome]

👋 Thanks for opening your first issue here! 👋

If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. We get a lot of issues on this repo, so please be patient and we will get back to you as soon as we can.
To help make it easier for us to investigate your issue, please follow the contributing guidelines.

[gesinger]

Hey @evanfarina , thanks for reporting and for the stream link. We took a look at the stream, and, as you said, the PTS values of the audio and video segments are different.

One question I had is: are the audio and video streams synced such that...

(1) each should be played from their respective starts (i.e., treat video PTS of 25.385s as time 0s and audio PTS of 3.392s as time 0s)?

(2) video PTS 25.385s should be treated as time 0s and audio should start at PTS 25.385s

(3) audio PTS 3.392s should be treated as time 0s and the stream should have a gap between 3.392s and 25.385s (where there is missing video content)

I ask because our latest playback engine release, VHS 2, fixed some issues related to syncing audio and video content present in VHS 1, and this may have caused the stream to stop working. Previously, we adjusted media content timing, but that ended up leading to desyncing of some other content.

If the stream here follows (1), then this is likely the issue. If the stream follows (2) or (3), then we should be able to address it, but not sure when we'll get around to it (and, in any case, we would recommend aligning the content to prevent gaps and wasted bandwidth).

But regarding (1), which I believe may be the case here, from my reading of the HLS spec, the content should be changed:

Matching content in Variant Streams MUST have matching timestamps.
      This allows clients to synchronize the media.

https://tools.ietf.org/html/draft-pantos-http-live-streaming-23#section-6.2.4

And demuxed audio streams here count as variant streams:

 The EXT-X-MEDIA tag is used to relate Media Playlists that contain
   alternative Renditions (Section 4.3.4.2.1) of the same content.  For
   example, three EXT-X-MEDIA tags can be used to identify audio-only
   Media Playlists that contain English, French and Spanish Renditions
   of the same presentation.

https://tools.ietf.org/html/draft-pantos-http-live-streaming-23#section-4.3.4.1

and

The rules for Variant Streams also apply to alternative Renditions -
   see Section 4.3.4.2.1.

https://tools.ietf.org/html/draft-pantos-http-live-streaming-23#section-6.2.4

In addition, I noticed that despite the different PTS values, each playlist has an EXT-X-PROGRAM-DATE-TIME that match (are equivalent in value despite the different media timings), which seems to go against the recommendations:

The server MAY associate an absolute date and time with a Media
   Segment by applying an EXT-X-PROGRAM-DATE-TIME tag to it.  This
   defines an informative mapping of the (wall-clock) date and time
   specified by the tag to the first media timestamp in the segment,
   which may be used as a basis for seeking, for display, or for other
   purposes.

https://tools.ietf.org/html/draft-pantos-http-live-streaming-23#section-6.2.1

If any Media Playlist in a Master Playlist contains an EXT-X
      -PROGRAM-DATE-TIME tag, then all Media Playlists in that Master
      Playlist MUST contain EXT-X-PROGRAM-DATE-TIME tags with consistent
      mappings of date and time to media timestamps.

https://tools.ietf.org/html/draft-pantos-http-live-streaming-23#section-6.2.4

Let me know which case the streams are synced by though, and we can go from there.

[evanfarina]

Hi @gesinger , For your first question, we expect option #2: video PTS 25.385s should be treated as time 0s and audio should start at PTS 25.385s.

I'll follow up soon re. your second question

[evanfarina]

Hey @gesinger - For my own understanding, do you know which section in the Spec speaks to how the player should interpret PTS values (option 1-3 in your comment above)? I took a look through today but am having trouble finding what I'm looking for. I'm curious if both option #1 and #2 are valid and whether we should support each simultaneously or if one of the options is clearly desired and the others are considered bugs.

[gesinger]

I don't think it's explicitly listed in the spec because the case is unexpected for media content. Since the media is supposed to be presented in the order of its timestamps, and since audio and video should follow the same order of timestamps, what is appropriate is generally to consider the first timestamp of content as time 0. In this case, since audio starts first, at time 3.392s, it would be the start of content. However, since we expect there to be both audio and video content, and don't fill gaps in video content (we do fill small audio gaps, but that is a workaround and wouldn't be used here), video's start of 25.385s should be time 0 in our player, and the first ~22s of audio content skipped.

Option 1 would lead to desyncing of much well-formed content.

Option 2 should be the route we take for this player (since we don't fill gaps of missing video with a blank screen or stretched keyframe).

[gkatsev]

I just tested the new stream on a bunch of MSE based players and unfortunately, none of those players work. Interestingly, most other players refuse to play the stream altogether and you're not even able to seek forward in the stream and get any playback.
In addition, when testing in Safari, I noticed the audio cut off for a second at the 20s mark. We've seen cases like this before where no MSE-based player is able to play the stream but it works in Safari. Unfortunately, Safari's playback engine is very robust and can recover from a lot of issues that we cannot do in an MSE based engine.
It's hard to justify trying to fix a stream that isn't able to be played back anywhere except Safari.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[RafalLukawiecki]

Still an issue

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[RafalLukawiecki]

Unless it is fixed, closing bugs by marking them stale is a waste of everyone’s effort, and discourages reporting.