A method to stitch together VOD clips

[joeyparrish]

It might be interesting to allow an application to stitch together multiple VOD clips.

For example:

player.load('/vod.mpd').then(function() {
  return player.appendVod('/other_vod.mpd');
}).then(function() {
  return player.appendVod('/hls_vod.m3u8');
});

The periods of the new content would be appended to the periods of the old content, and the presentation timeline would be extended to match.

There would be no requirement that the same manifest parser be used for each clip, so DASH could be appended together with HLS.

Appending a manifest which turns out to be live or IPR content would fail with an error such as CANNOT_APPEND_NON_VOD_CONTENT.

If the initial load() call is for live or IPR content, the appendVod() call would call fail with an error such as CANNOT_APPEND_TO_NON_VOD_CONTENT. (Subtle difference in error code.)

If filtering the new content against the existing content leaves no playable streams from the new content, the call would fail with an error such as CANNOT_APPEND_INCOMPATIBLE_CONTENT. For example, appending TS content after MP4, or appending H.264 after VP9.

If this capability would be interesting to you, please comment and let us know!

[MichaelZaporozhets]

We implemented this by building a 'stream merging' api that normalises and concatenates mpd sources into a single stream available as hls & dash on the fly and passing that as the source for each.

The code is a bit of a mess but it works well. No reason the input can't be amended to support hls streams too.

.following

[birme]

On a side note: This could also be achieved server-side to avoid having to re-implement this on every device platform. Something we developed as a proof-of-concept and made available as open source: https://eyevinn.github.io/channel-engine/

With that said, the proposed method would still be cool and nifty feature to have in Shaka.

[osmestad]

I think this would be very interesting for our use case (I work on music streaming at TIDAL) currently we use two instances of Shaka player to do preloading of the next track when the playing track is close to finished. Ideally we would like to have as short a gap when switching between tracks as possible (gapless being the goal) :-)

[enjikaka]

@joeyparrish (in reply to #880 (comment)); Cool. Like Øyvind said gapless is the goal for us at TIDAL. I'm free to look into this and would like to help if possible.

Any pointers were I should start digging in the shaka code base?
Any plans or ideas for this feature other than the above mentioned appendVod method?

[joeyparrish]

In v3.0, we introduced a utility that stitches together DASH periods. That could also be adapted for used in appendVod.

The preconditions can be checked in manifest.presentationTimeline. If presentationTimeline.isLive() || presentationTimeline.isInProgress(), then appendVod() should fail.

The most complicated part will be starting a secondary "load graph" to parse the new manifest. (This is effectively a state machine that implements all the various parts of load() and interactions with attach(), detach(), unload(), and destroy().) In this way, it does connect to #880, since preload may also want to refactor some of the load graph stuff to be able to start the load process for multiple pieces of content in parallel.

In Player (lib/player.js), in shouldUseSrcEquals_, you'll see how we decide if something should be played with src=, as opposed to MediaSource. If the new content should be src=, appendVod should fail. If the old content is using src=, appendVod should fail then, too. (If this.loadMode_ == shaka.Player.LoadMode.SRC_EQUALS)

In Player, in onInitializeParser_, you can see some of the code to select a manifest parser and initialize it. The next stage is in onParseManifest_. These are both parts of the load graph, though you may want to try to read through and study the whole thing. I haven't spent much time thinking about what parts are needed for preloading/appending a manifest.

Finally, you'll want to look at lib/util/periods.js. You construct a shaka.util.PeriodCombiner and call combinePeriods, passing information about two or more "periods". That's in quotes because in DASH, those are actual periods, but here, you'd have one fake period representing the existing manifest and one representing the new content manifest. When combinePeriods is done (await it, since it's async), you can call getVariants() and getTextStreams() on it to update the manifest held by the Player. You may also need to fire an event to show that tracks have been updated.

That's a very rough sketch of the components I expect should be involved, but we haven't done any serious design work on this or thought through all the changes or their potential side-effects.

It looks to me like a very big change. So you should feel free to try making some changes as a proof-of-concept, but it might save us all time and effort in the long term if we have more design discussions as you progress. If the first we see of your chosen approach is a completed PR, it might be a painful review, and any design changes we ask for at that late stage may be more difficult to make.

Thanks for expressing interest in working on this!

[enjikaka]

Thanks for the thorough reply @joeyparrish! I'll have a look at this methods and get a bit more acquainted with the code base. It looks like preload is something the main shaka team is going to jump on soon looking at the roadmap and wishes for next release? Maybe I should wait with tying to implement gapless - or at least the loading aspects - until that is done?

[avelad]

Is anyone interested in this functionality?

[enjikaka]

Is anyone interested in this functionality?

Yes, 257 %! Needed for gapless playback :)

[theRealRobG]

Hi all,

I'd like to add that it would probably be worth considering how the solution extends to also support stitching from VOD to live. This is something that is already declared as possible in ISO/IEC 23009-1 5.11.2 Regular Chaining:

MPD chaining can for example be used for pre-roll ads or creating a sequence of programs using multiple MPDs. In this case, the chained-from MPD may be of type static, whereas the chained-to MPD may be of type dynamic. In this case, the client is expected to join the dynamic MPD at the live edge, or if an anchor is presented as defined in C.4, at the indicated time in the anchor.

Not to say that Shaka needs to support all supplementary/essential descriptors out there, but just to emphasize that there is precedence for a need to support more than just VOD to VOD, so something to take into consideration when solving for VOD to VOD.

[joeyparrish]

Okay, makes sense. If the additional content is live, that should be okay. (We will have to adjust timestamps in the live content to make it happen.) But the existing content must be static to be able to append anything new.

Does that sound good?

[theRealRobG]

Yes, makes sense to me.

The document does mention that "the chained-from MPD may be of type static or dynamic", but I have to think that surely that can only apply to a dynamic manifest that has turned static, or maybe just a typo... Because I don't even understand the use case otherwise (when should the player transition over?). I'll ask in the video-dev #dash channel to be sure... Perhaps Alex Giladi can shed some light on that.

In any case, our use-case only considers existing content being static. We would consider this for live pre-roll, which would be represented as some small chain of static assets (e.g. 3), followed by a dynamic asset starting at the live point.

[theRealRobG]

Actually, reading the spec again, I can see how earliestTimeToResolve can be used as a means for giving an indication to the client on when to transition to the next content even while the current content is still dynamic... I've asked in video-dev#dash to clarify.

And actually, it occurred to me that we do have a use-case today around this, where we support live binge via an in-stream marker that declares the "end" of the content (e.g. shortly after final whistle of a football match) even before the actual live stream concludes, so as to move users to other related content.

That being said, I don't feel strongly about supporting that use-case via a stitching solution. I can see how that would be quite complex to implement and even the user experience may be quite jarring (I think it would feel more natural if there was at least some loading time / break between the live content and where you binge to).

For me the restriction of existing content being static for this feature makes sense 👍

Was just pointing out above that theoretically it seems the spec allows for a dynamic -> dynamic transition.

Update: I got an answer from video-dev#dash. earliestTimeToResolve does not indicate when client should transition to next content, instead, it provides an indication on when the client should resolve the next chained to MPD (nothing to do with actually transitioning to it). Nevertheless, there is a way that an MPD@type="dynamic" asset can terminate even without transitioning to static, which is via the use of the urn:mpeg:dash:event:ttfn:2016 event scheme (either inband or MPD event), as described in 5.10.4.6 Presentation Termination Event of ISO/IEC 23009-1 ed.5. It was indicated that sometimes this is desirable because the service provider may not want to indicate that the manifest is now static as segments may not be available to present anymore, so instead they may choose to keep the MPD dynamic so that the client loads at the live edge, and is immediately notified that the event has already concluded.

@joeyparrish when you mention "the existing content must be static to be able to append anything new", do you mean that you wouldn't consider an API that would allow appending to something that is currently live but could end and then transition? So something like "scheduling an asset to be appended when the current asset end is known".

Again, I'm not pushing for it, just curious on your thoughts.