Support Video Chapter in MediaMetadata

[jiajiabingcheng]

こんにちは TAG-さん!

I'm requesting a TAG review of the Chapter Information in MediaMetadata.

Background:
Media content such as youtube videos can contain chapters or markers that break down the video’s content by timestamp. There is currently no concept of chapter information within the MediaMetadata to capture such information. See the proposal: w3c/mediasession#273

In this project we add the ChapterInformation attribute in the current MediaMetadata.

What changed in the explainer introduction section:
Before this change:
A MediaMetadata object can contain media metadata like title, artist, album and artwork.
After this change:
A MediaMetadata object can contain media metadata like title, artist, album, artwork, and video chapter information.

• Explainer¹ (minimally containing user needs and example code):
  https://github.com/w3c/mediasession/blob/main/explainer.md

• Specification URL:
  https://www.w3.org/TR/mediasession/#the-chapterinformation-interface

• Tests: [wpt folder(s), if available]
  https://chromium-review.googlesource.com/c/chromium/src/+/5516503

• User research: [url to public summary/results of research]

• Security and Privacy self-review²:
  The current MediaMetadata does not provide any new data to the website. It just adds a ChapterInformation attribute to the existing MediaMetadata that the website can use to set video chapters. See more in the security and Privacy section in the Chrome Status

• GitHub repo (if you prefer feedback filed there):
  N/A

• Primary contacts (and their relationship to the specification):
  Jiaming Cheng (@jiajiabingcheng), Google

• Organization(s)/project(s) driving the specification:
  Google, MediaSessionChapterInformation

• Key pieces of existing multi-stakeholder (e.g. developers, implementers, civil society) support, review or discussion of this specification:

  • Chromium comments: [url]
  • Mozilla comments: The ChapterInformation interface mozilla/standards-positions#1019
  • WebKit comments: Support Video Chapter in MediaMetadata WebKit/standards-positions#344
  • Etc.

• External status/issue trackers for this specification (publicly visible, e.g. Chrome Status):
  https://chromestatus.com/feature/6682585059295232?gate=5003115407605760
  Further details:

• I have reviewed the TAG's Web Platform Design Principles

• Relevant time constraints or deadlines: [please provide]

• The group where the work on this specification is currently being done:

• The group where standardization of this work is intended to be done (if current group is a community group or other incubation venue):

• Major unresolved issues with or opposition to this specification:

• This work is being funded by: Google

You should also know that...

[please tell us anything you think is relevant to this review]

---

CAREFULLY READ AND DELETE CONTENT BELOW THIS LINE BEFORE SUBMITTING

Please preview the issue and check that the links work before submitting.

In particular, if anything links to a URL which requires authentication (e.g. Google document), please make sure anyone with the link can access the document. We would prefer fully public documents though, since we work in the open.

¹ We require an explainer to give the relevant context for the spec review, even if the spec has some background information. For background, see our explanation of how to write a good explainer. We recommend the explainer to be in Markdown.

² A Security and Privacy questionnaire helps us understand potential security and privacy issues and mitigations for your design, and can save us asking redundant questions. See https://www.w3.org/TR/security-privacy-questionnaire/.

³ For your own organization, you can simply state the organization's position instead of linking to it. Chromium doesn't have a standards-positions repository and prefers to use comments from the teams that maintain the relevant area of their codebase.

[hober]

We already have a mechanism to expose timed metadata—track elements, WebVTT, etc. Can't we just use that existing infrastructure?

[hober]

Also, this should probably be exposed in the same way as the strobing mitigation stuff.

[jiajiabingcheng]

Hi @hober,

Thanks for your feedback!

Before starting implementation, we considered using the Web API to set chapter information via elements (https://developer.mozilla.org/en-US/docs/Web/HTML/Element/track) just as you suggested. However, after discussing with our Media Session and YouTube teams, we identified several drawbacks to this approach:
1, Lack of image support: There's no standardized way to include chapter images using this approach.
2, Core player integration: It would require modifications to the core player code since it touches the media element. This would be inconsistent with how the rest of the system UI uses MediaSession, potentially leading to maintenance challenges and inconsistencies in the future.
3, Awkward DOM usage: Representing structured chapter data directly within the DOM felt cumbersome and not ideal for this use case.

And this proposed approach, adding a new ChapterInformation field to the current metadata, can help ensure we have a well defined API and maintain a simple implementation process for the websites.

Let me know if you have any questions or concerns :]

[cookiecrook]

@jiajiabingcheng you may be interested in the progress on WebVTT kind disambiguation, including metadata:

• VTT Metadata Cue format is ambiguous; some metadata may be unintentionally presented to the user in a context outside HTML w3c/webvtt#511
• New ATTRIBUTES block w3c/webvtt#523 (PR updated today)

[hober]

I had a good conversation with @jernoble about this a few weeks ago. It does make sense for this to be included in the Media Session API. (The API addresses different use cases, like remote controls or lock screen controls, that would be ill-suited for WebVTT.)

[hober]

Thanks for bearing with us. This looks good to us.