mse-in-workers-using-handle-explainer.md

Media Source Extensions: MSE-in-Workers Explainer

Author: Matthew Wolenetz, Google Inc. - December 17, 2018. Last update July 27, 2022.

Updated Oct 29, 2020 to remove handle, then updated July 27, 2022 to add the MediaSourceHandle back (API discussions in Media WG evolved during this feature's specification and experimental implementation development in Chromium.)

This updated version also is limited to expanding MSE usage beyond the main thread to just a DedicatedWorker context (not also a SharedWorker context.) Further, this updated version drops the 'early-open' portion. These were removed as major simplifications, and are not precluded from being proposed again in future (especially the 'early-open' functionality.)

tl;dr

We propose enabling Dedicated Web Workers to be able to create MediaSource objects and obtain a MediaSourceHandle object from each MediaSource. They may then transfer the MediaSourceHandle to the main thread for use in attaching to media elements. The direct manipulation of the MediaSource and ancillary objects like SourceBuffer is allowed only from the context that originally created the MediaSource object. This allows both:

• improved access to, and
• improved performance of the Media Source Extensions API (MSE).

We initially incubated this idea in the WICG, then moved specification development into the W3C MSE API specification with the Media Workgroup while experimental implementation and trials were done in Chromium.

Implementation status as of last update

• Chrome - Preparing to ship. Implementation is in Chromium as of 105.0.5180.0, behind experimental flags.
• Firefox - interest expressed in F2Fs, interacted and gave support and great feedback during development of feature specification, especially on two PR's 305 and 306.
• Safari Technical Preview - interest expressed in F2Fs and in Media Workgroup meetings, interacted and gave support and feedback relative to PR 305 and 306, too.
• Current experimental web-platform-test results:
  • Tests implemented and passing in experimental Chrome runs wpt.fyi link

Demo of Usage to Avoid "Buffering Jank" When There's Main Thread Contention

• See the demo source and instructions in its repository. Note, prior to July 27, 2022, this demo exercised the version of the implementation and specification that used legacy MediaSource object URLs for attachment of Dedicated Worker MediaSource objects. The specification and experimental implementation changed to use transferable MediaSourceHandle assigned to media element srcObject attribute for attachment of worker MediaSource objects in July 2022 (in Chromium 105.0.5180.0 and as of spec PR 306 being merged.

Background

Web authors have consistently requested that MSE be available from Worker contexts: MSE spec issue.

Separately, we have strong suspicions that user experience metrics, such as those measuring how long media playback takes to start, can be impaired by MSE REC spec requiring implementations to delay operations like addSourceBuffer() until after MediaSource is attached: Chromium's Tracking bug for MSE sourceopen latency improvement work.

In the absence of such mechanisms, web authors are forced to await MediaSource attachment completion on the main Window context, and perform all operations on the MSE API also on the main Window context. Especially when the main context is busy servicing other application logic, these restrictions can significantly impair the user experience. For example, even if an app uses Worker contexts to fetch media to feed to MSE, such feeding must still occur on the main context, resulting in delayed media playback start and recurring playback stalls in extreme cases when the playback reaches points in the presentation timeline for which the app has been unable to buffer enough media.

Design consideration was given to alternative ideas that are not in scope of this proposal:

• Upgrading the existing MediaSource object to be Transferable. However, EventTargets like MediaSource cannot be transferred. A viable, ergonomic way of overcoming this appears unlikely.
• Upgrading HTMLMediaElement (and its many related objects and APIs) to be usable on a Worker context. Though this might be a solution, the complexity of this approach led us to propose a simpler change scoped to just MSE.
• Using legacy MediaSource object URLs assigned to a media element's src attribute (or used in a media element's <source> tag) for attaching Dedicated Worker MediaSource objects (instead, this design provides a new, transferable MediaSourceHandle object in lieu of a MediaSource objectURL to enable attachment via the srcObject attribute on an HTMLMediaElement.)
• Enabling usage of MSE API from SharedWorker or ServiceWorker contexts.
• Providing an explicit method, such as MediaSource.open() to enable applications to begin adding SourceBuffers and starting initial appendBuffer() operations on them prior to the attachment of the MediaSource to an HTMLMediaElement completing.

Note that this proposal does not preclude separate or subsequent effort to pursue these other ideas.

Proposed Plan

This proposal is focused on enabling DedicatedWorker usage of MSE API to improve application ability to buffer media for smoother playback despite a potentially busy main Window context. It does not change the pre-existing usability of the MSE API on the main Window context, for usage with MediaSource objects created on that main context.

The REC version of Media Source Extensions API (MSE) allows access to the API only on the main document Window context. Further, the MSE API requires a MediaSource instance to first be "attached" to an HTMLMediaElement for that element to be extended by MSE and for the MediaSource to become open and allow further operations like SourceBuffer creation and manipulation. The "attachment" is started by creating a MediaSource objectURL, and then setting the src attribute of the HTMLMediaElement to be that objectURL.

To maintain backwards compatibility with that behavior, while affording improved MSE API access and performance, this proposal adds a feature-detectable MediaSource constructor access to DedicatedWorker contexts. It also adds a readonly getter handle on MediaSource objects to obtain a MediaSourceHandle object from a MediaSource object. This MediaSourceHandle object is added by this proposal. A MediaSourceHandle object is Transferable, and can be used to attach the underlying MediaSource to an HTMLMediaElement by setting the HTMLMediaElement's srcObject attribute to the MediaSourceHandle object. Unlike legacy MediaSource object URLs, there is precisely one MediaSourceHandle object for each MediaSource object, and that handle can be used at most once for attachment to an HTMLMediaElement (as a simplification.)

To prevent leakage of access cross-context by the app to the MSE object model, care will be taken especially to modify what the Audio,Video,TextTrack reports in the extended-by-MSE sourceBuffer attribute depending on which context is accessing the object: if not the same context as where the MediaSource was constructed, then that field will return null.

Open Questions

Can implementations of this proposed API significantly reduce the performance problems and eliminate the API access problems versus REC MSE?

Initial Thoughts:

This is a big part of why we're incubating. Of course, standardizing a more ergonomic API with improved interoperability are also strong reasons for incubation.

Why must we create the MediaSource first in a DedicatedWorker context instead of using something, like the MediaSourceHandle, constructed on the main Window context, and extracting something transferable from it to transfer to other contexts if needed there?

Thoughts:

While this might allow some parallelization of attachment on the main document Window context while the DedicatedWorker context is still starting, we're not convinced yet that the additional complexity is worth it. For example, using subworkers or other multi-worker arrangements communicating directly via the Channel Messaging API to reduce the granularity of priority worker startup might be a reasonable workaround. This is another good reason to incubate. Note, such is not precluded by this proposal.

Can detachment of MediaSource not automatically shutdown all SourceBuffers (and remove all their buffered media?)

Initial Thoughts:

Perhaps, but an orthogonal feature proposal to let an app explicitly request a MediaSource object to behave differently on detach of it would reduce the complexity of each proposal and let them be developed in parallel or separately.