Adding a registration for H.263

[Yahweasel]

This PR addresses my own #416, and adds a registration for H.263. Included in the registration is all revisions of H.263 by ITU-T, and for the latest revision (v3, H.263++), the profiles and levels specified in Annex X. I believe it covers all realistic use cases of H.263 excluding MPEG-4 Part 2, which should be covered by a separate registration if at all.

[Yahweasel]

Just to confirm for the workflow step, this contribution affects only the non-normative codec registry.

[sandersdan]

Are levels in H.263 inter-compatible, as in, can an encoder output a lower level than requested like we have defined for H.264?

[Yahweasel]

With the exception of level 45, each level is strictly looser than all previous levels, so an encoder outputting at, e.g., level 10 is correctly outputting at level 20 or 30 or 40. 45 is an exceptional case, and while all streams in level 45 are also valid bitstreams in all higher levels, it is not true that all valid bitstreams at lower levels are valid as level 45. Because 45 is a weird exception, but all other levels are inter-compatible, I think it's easier to leave that as implicit, rather than repeating the document's rather long-winded explanation of their intercompatibility that I've just detailed here :)

[tidoust]

There are two parts in this pull request:

1. It adds a registry entry to the registry. The requirements are described in the registry and all seems good to me here.
2. It creates a WebCodecs registration document for H.263. Adding that document to the repo means that you're asking the Media Working Group to take responsibility for the document's content, publication and maintenance. We need to make sure that the Media Working Group is ok with that before we proceed. Cc @chrisn @jernoble

[Yahweasel]

2. It creates a WebCodecs registration document for H.263. Adding that document to the repo means that you're asking the Media Working Group to take responsibility for the document's content, publication and maintenance. We need to make sure that the Media Working Group is ok with that before we proceed. Cc @chrisn @jernoble

My assumption is that this is how it's supposed/intended to work, if for no other reason than that every other codec registration is also maintained in this repository. If it's more reasonable to put the codec registration document elsewhere and simply link it from the codec registry, I'm happy to do that; perhaps it would even be good to be the first one, as it would be the example of how to register a codec externally.

That being said, H.263 is set in stone, so notionally any maintenance is just to do with changes to the information provided by registration documents. Hopefully it's not much of a burden.

[chrisn]

We need to make sure that the Media Working Group is ok with that before we proceed.

Yes, we need consensus from the Working Group to add this registration. To help us make that decision, can you describe or share a link to your WebCodecs polyfill? Is it open source?

[Yahweasel]

We need to make sure that the Media Working Group is ok with that before we proceed.

Yes, we need consensus from the Working Group to add this registration. To help us make that decision, can you describe or share a link to your WebCodecs polyfill? Is it open source?

Of course it's open source!

https://github.com/ennuicastr/libavjs-webcodecs-polyfill

LibAVJS-WebCodecs-Polyfill uses libav.js, which is a port of ffmpeg (via emscripten) I've been maintaining for various projects for a couple years. Since the underlying software is ffmpeg, most of this is automatically on solid footing; I've got the vast majority of the WebCodecs API implemented on the first day I started polyfilling it.

[chcunningham]

My assumption is that this is how it's supposed/intended to work, if for no other reason than that every other codec registration is also maintained in this repository.

I think you're assumption is correct and @tidoust is just describing the next step in this process. IMO we don't have to block landing your PR on working group review. It could remain an "editors draft" until WG votes to formally add the registration. @tidoust lmk if I have that right.

[chcunningham]

I typically see codec strings break down by profiles rather than annexes. I appreciate that profiles came along pretty late here, but I wonder if they could still serve this purpose and thereby simplify the codec string scheme. At a glance, it looks like the profiles refer to combinations of features from earlier annexes. Would it work to simply have a string like h263.[profile]?

[Yahweasel]

The problem is that the profiles don't have a clean relationship with the revisions, but practical codecs are by revision. Profile 0 is only decodable with H.263v1, but doesn't use all of H.263v1's features. Profiles 1–3 are decodable with H.263v2, but don't use all of its features. Profiles 4–8 all require H.263v3. Any practical implementation is going to use off-the-shelf codecs, and in practice, the codecs are by revision. For instance, ffmpeg has "h263" (v1) and "h263p" (v2).

[chcunningham]

I found one mention of "s263" in https://cconcolato.github.io/media-mime-support/. @cconcolato can you shed light on where that comes from and any thoughts on the scheme proposed herein?

[Yahweasel]

Looks like s263 is used for 3GPP, which is one of many "H.263 but..." codec specifications, as is MPEG-4 Part 2. I must admit, I'm wholly unaware of what the "but" part is; it may just be a particular restriction of features.

[chrisn]

3GPP TS 26.244 defines the s263.profile.level codecs string format for H.263++. I wonder if we can reference this for H.263++?

[Yahweasel]

I would be happy to add the s263.profile.level strings as alternatives, but as mentioned in one of the threads here, the tricky thing with the H.263(+(+)) profiles is that they don't have any clean correspondence to the document revisions, but actual codecs implement document revisions, so not having a way to specify H.263 vs H.263+ vs H.263++ isn't really an option.

[chcunningham]

nit: this doc has a lot of "may" and "must", which are unfortunately reserved for normative documents (telling implementers what they must/may do). We have to find alternative phrasing (ex: #340)

[Yahweasel]

I didn't cap and em them because I assumed that was reserved for normative documents. Are "should" and "ought" reserved?

[padenot]

The list is herre: https://datatracker.ietf.org/doc/html/rfc2119. should is reserverd, ought is not.

[Yahweasel]

"Can" and "needs to" it is ;)

[chcunningham]

@Yahweasel just to be clear, is it your intention to be on the hook for future maintenance (PR reviews) for this doc?

@tidoust, any issue with that? For my part, happy to have the help, but I don't know relevant W3C rules offhand.

[tidoust]

There is an issue in that, per process, "an editor must be a participant". I'm confident that we can find a suitable workaround though.

@Yahweasel could perhaps be granted an invited expert status, although note that has some licensing implications (the Patent Policy does not apply to this document, but the invited expert status is linked to the group as a whole, and the group develops normative specs), or simply switch to another official spec editor.

[Yahweasel]

I would warn you that I'm not, in particular, an expert. I guess I'm an expert in the way that anyone is an expert (I fell into the digital media field and learned while falling), but even as such, my expertise is far more in audio than video. This whole rabbit hole is scratching a particular itch for another project. I intend to maintain that project, of course, and to that end, this is part of that. I don't particularly mind being on the hook for this document, as I expect that (a) there's not much maintenance involved for a codification of a codec that hasn't changed in 15 years, and (b) no one else cares about H.263 :) .

[tidoust]

I think you're assumption is correct and @tidoust is just describing the next step in this process.

Absolutely. Sorry if these process steps feel a bit impersonal. By definition, they are ;)
I personally agree that collecting all registrations in one place under the umbrella of the Media Working Group is a good thing.

IMO we don't have to block landing your PR on working group review. It could remain an "editors draft" until WG votes to formally add the registration. @tidoust lmk if I have that right.

Different groups have different perspectives. In some groups, participants feel strongly that whatever lands in the repo needs to have been vetted by the group. Other groups take a more casual approach, sometimes flagging drafts as "Unofficial" (status UD in Bikeshed) to distinguish them from Editor's Drafts. All such drafts typically originate from group participants, which is not the case here, so I would recommend running the call for consensus before landing the PR.

[chrisn]

I would prefer to see WG consensus before merging. We have a WG call next week, so can include in the agenda.

[Yahweasel]

I really don't mind one way or the other how things go. I decided I should ask about H.263 in the registry because I intend to use H.263 with my own polyfill, and I'd rather that not be a special exception, so that notionally a future implementation could be compatible. If this is all too messy, then I'll just have it be a special exception in my polyfill.

[Yahweasel]

I should also say: I don't even intend to implement H.263v3, as FFmpeg doesn't. H.263v3 is very uncommonly used, because it essentially competes with MPEG-4 Part 2, which isn't quite compatible. I included H.263v3 and its profiles and levels because it felt like it would be really bizarre to codify H.263 but not codify its latest version. This is also why the profiles and levels are quite marginal; practical implementations don't support them, because practical implementations are of H.263v2 or v1.

[chrisn]

Discussion from today's working group meeting: https://www.w3.org/2021/12/14-mediawg-minutes.html#t02

[chrisn]

Sorry for the delay. The call for consensus passed with no objection, so next step is to invite some more review feedback.

https://lists.w3.org/Archives/Public/public-media-wg/2022Mar/0002.html

[chrisn]

We have received some review feedback, which said:

• RFC 4629 contains media types for video/H263-1998 and video/H263-2000. The proposed WebCodecs registration refers to the 1998 version of H.263 as H.263v2 and the 2005 version as H.263v3, but does not seem to consider the 2000 version explicitly, even though that version is a specific focus in RFC 4629 (and RFC 2429 before it). That seems a little odd, especially in regard to the status of Annexes U and V. Maybe there should be v3 and v4 names to distinguish between those, and maybe suffixing names by years would be more clear than versions. Year suffixes were used in RFC 4629.

• There is some commentary about assuming decoders support all optional features. This seems rather optimistic and generally impractical. Some of the options may require special system support, for example.

• The latest annex, Annex X, defines profiles and levels of H.263. Specifying packages of capabilities in terms of those profiles and levels would probably be a better idea, as those were agreed by an approved consensus as part of H.263 itself.

[Yahweasel]

Hello,

The 2000 "version" of H.263 isn't a single document. It's the 1998 version with three annexes (and an informative appendix). I wanted to avoid specifying annexes and stick to published documents. The only addition in the 2005 version relative to the 2000 "version" is Annex X, which RFC 4629 treats as part of the 2000 version anyway, in spite of the fact that Annex X was published in 2001. I'm referring to the 2005 version as H.263v3 (or H.263++) because it's a cohesive document, rather than the previous document with some annexes. The actual specifications referenced are identical.

You're right about decoder support. I'm not sure what I ought to specify though. I guess this relates to the following thoughts.

I have indeed specified Annex X's profiles and levels, but yes, only as part of v3. Unfortunately, they're not as useful as you may think. Popular open source en/decoders of H.263, in particular ffmpeg, don't use them (because open source en/decoders only implement H.263+, not H.263++), and they're not designed to correlate well to H.263 releases. Only profiles 0–3 are restricted to H.263v2 features, but there are H.263v2 features that are only “activated” in profiles 6 and higher, along with H.263v3 features.

I think I could use Annex X profiles practically as an intersection with releases. So, version 2 profile 8 means profile 8 except for its annex U feature (and version 1 doesn't use profiles since it's always profile 0). This could also resolve the troublesome "must support all" gobbledygook.

[chcunningham]

@chrisn, just checking in. What are next steps?

[chrisn]

Thanks Chris. Is there any update we can make to the registration based on the latest replies above (particularly the discussion of Annex X profiles and levels)?

If we need further guidance, Gary Sullivan suggested that we could bring this to ITU-T SG-16. To do that, I would suggest that we pull together a summary and a set of specific questions. We can then either approach Gary again directly or send as a liaison to ITU-T. If we do that, it may be good to combine with questions about the proposed H.265 registration, to cover everything in one go.

[chrisn]

We discussed in the July 12 Media WG meeting.

[chcunningham]

If we can use profiles, that has my vote. It matches what's done for other codecs and it pushes the decision about how to group features to ITU folks who are certainly the experts. I understand the idea to combine profiles with versions, but that seems too complex. Let me poke a bit more to see if we can reconcile profiles w/ implementations.

Popular open source en/decoders of H.263, in particular ffmpeg, don't use them (because open source en/decoders only implement H.263+, not H.263++), and they're not designed to correlate well to H.263 releases. Only profiles 0–3 are restricted to H.263v2 features, but there are H.263v2 features that are only “activated” in profiles 6 and higher, along with H.263v3 features.

I propose that WebCodecs implementers act as a bridge between the versioning schemes. For example, if you're underlying codec supports H.263v2, return supported=true for profiles 0-3. If the underlying codec supports a subset of required features for some higher level profiles (e.g. H.263v2 vs profile 6), I propose that we return supported=false for such profiles. This may mean that we don't advertise the full list of supported features (choosing instead to break at the granularity of profiles), but I imagine that still strikes the right balance between power/simplicity.