Support for unknown boxes

[cconcolato]

The BSF spec says:

Valid top-level boxes such as pdin, free, and sidx are allowed to appear before the moov box. These boxes must be accepted and ignored by the user agent and are not considered part of the initialization segment in this specification.

and

Valid top-level boxes defined in ISO/IEC 14496-12 other than ftyp, moov, styp, moof, and mdat are allowed to appear between the end of an initialization segment or media segment and before the beginning of a new media segment. These boxes must be accepted and ignored by the user agent and are not considered part of the media segment in this specification.

In my testing, only Safari supports boxes like unkn. All browsers support a free box though. It should be clarified what "Valid top-level box" means. Does it mean an actual box defined somewhere, and if so where? or does it mean any box like unkn should be supported as long as it is well-formed.

[wolenetz]

We discussed this during the Sep 28 Media WG teleconference (https://www.w3.org/2021/09/28-mediawg-minutes.html).

My understanding of the intent of (and Chromium's implementation of) "Valid top-level box" means what is defined in the normatively referenced [ISOBMFF] spec (http://standards.iso.org/ittf/PubliclyAvailableStandards/c068960_ISO_IEC_14496-12_2015.zip) at the top-level of the box parsing hierarchy.

While not clear in the BSF spec, my further understanding of the intent of handling other boxes encountered at the top-level, either valid but not top-level, or not defined in that [ISOBMFF] spec, was to trigger the append error algorithm.

During the teleconference call, I don't think we arrived fully at a conclusion for the next steps:

1. I think we were in agreement that any non-top-level box that is defined in [ISOBMFF] should trigger the append error algorithm.
2. How to handle boxes that are not defined in that [ISOBMFF] that are encountered at the top-level was not fully settled:

• @jernoble noted that it seems like an implementation detail: could throw an error; but it's OK to skip an unknown box.
• @cconcolato noted that just consuming the unknown box and continuing parsing might be the better route, letting the spec and implementations scale (otherwise, each time [ISOBMFF] spec changes what is defined as a top-level box, implementations would also need to change). While unkn is an example here, @cconcolato also mentioned recent introduction in newer versions of [ISOBMFF] of compressed boxes as a real example.
• During the meeting, I was trying to think through some considerations, but only landed on the possibility of unknown boxes instead signalling a potentially malformed stream, enabling earlier decode error versus letting the parser skip over a potentially large number of bytes in the stream. Then I noted the QuotaExceeded exception could at least eventually signal an error if too much was buffered information were needed by the the parser to successfully accomplish the skip of an unknown box.

Following the teleconference, I thought a bit more on this, and want to mention that, while skipping unknown boxes enables continued buffering of streams when new top-level (or other mid-level) boxes are introduced, such boxes (like compressed boxes) might introduce a requirement on the implementation to be able to support (not skip) them to be able to interoperably and correctly parse, buffer and play the media.

A large benefit of immediately signalling error on unknown boxes encountered in the stream is precisely identifying implementation lack of support for the box. While it may scale less as [ISOBMFF] is updated, such updates have been rare.

So there is a tradeoff: scale and attempt to play streams muxed against some newer version of [ISOBMFF] skipping unknown boxes, or provide implementations and content providers more precise signals of what support is missing that requires interop updates (spec and implementation) to support the new streams. I tend to prefer the latter, though I believe this issue needs further discussion.

[cconcolato]

For reference, here is the list of top-level boxes defined in ISOBMFF (7th edition, WG03N0148_MDS19980) :

• ftyp File Type Box
• etyp Extended Type Box
• mdat Media Data Box
• free, skip
• pdin Progressive Download Information Box
• imda Identified Media Data Box
• moov Movie Box
• moof Movie Fragment Box
• mfra Movie Fragment Random Access Box
• meta Meta Box
• styp Segment Type Box
• sidx Segment Index Box
• ssix Subsegment Index Box
• prft Producer Reference Time Box

Note that there are other specifications (MPEG or not) defining top-level boxes. For example, the MPEG ones are:

• MPEG DASH defines the emsg box
• MPEG Partial File Format defines the fidx box (File Index Box)

[cconcolato]

@wolenetz In theory, yes, skipping a top-level box could potentially prevent you from interpreting correctly a next top-level box. In practice, ISOBMFF is extended in backwards-compatible ways. The top-level boxes used in MSE (moov, moof, mdat) have never been and will very unlikely be modified in a way that you need to understand a previous top-level box to interpret them correctly.