AVIF MIME type should either be required by web browsers or MIME Sniffing needs updating

[annevk]

As discovered somewhat late during JPEG XL review, it's currently not defined in the relevant web standards when something is AVIF. At least not in a way that matches what Chrome and Firefox ship, as I understand it.

Either AVIF's MIME type should be required and responses are only decoded as AVIF if the MIME type was supplied and is correct (this would require changes to HTML which defines image decoding infrastructure for the web for better or worse) or the algorithms in MIME Sniffing need to be updated to account for this new image format.

[cconcolato]

Quickly reading through the references indicated here, my understanding is that this issue is more for tracking what's happening in various other issues/specs, rather than for asking a change in the current specification. Am I correct?

[annevk]

Yeah, this is not a problem with the AVIF format itself, but I wasn't sure where else to raise it and it would be great if the AVIF specification community could take ownership in resolving it. (Happy to help though.)

[cconcolato]

I'm not sure it's possible to mandate that the MIME type be set correctly. It seems "easier" to update the MIME sniffing spec to make the "signature for MP4" take a parameter brandStart, renaming it "signature for ISOBMFF" and:

change the video part to say:

• If input matches the signature for ISOBMFF with brandStart "mp4", return "video/mp4"
• If input matches the signature for ISOBMFF with brandStart "iso", return "video/mp4"

(could add other brands, e.g. QuickTime files)

and add to the image part:

• If input matches the signature for ISOBMFF with brandStart "avif", return "image/avif"

(could add other brands for other HEIF-based formats)

[annevk]

The way we'd enforce the MIME type to be set correctly is by passing the MIME type to the decoder and the decoder would only continue down the AVIF path if the MIME type is correct (and ignores the MIME type for the other paths as per legacy behavior). (And also when navigating we'd only show the image if it has the correct MIME type and otherwise it would go down the download path.)

[jonsneyers]

ISOBMFF is also used by JPEG 2000, and can also be used by JPEG XL (though there's also a more compact header that is just the two bytes 0xFF0A), so updating the MIME sniffing spec to "know" about ISOBMFF seems like a good idea.

Enforcing the MIME type to be set correctly can be done, but I think it would be a significant hurdle for adoption of new codecs, since not everyone can actually control the MIME types sent by their server, and especially for new codecs, servers are likely to not signal a (correct) response type...

[baumanj]

@cconcolato's suggestion for updating mimesniff sounds eminently reasonable. While we're at it, it would be useful to implement a standard maximum number of bytes that will be sniffed for MP4 (ISOBMFF). Currently the spec puts that limit at 1445 bytes for sniffing generally, but both Firefox and Chromium use relatively arbitrary limits of 512 and 144 bytes respectively. Sniffing less is better, and the idea of a file with even 32 legitimately compatible brands strains credibility. I imagine a limit of 64 bytes (12 brands) would be reasonable, especially considering any files with more would just have to make sure they are setting the MIME type in the response header to eliminate sniffing as a restriction. That said, the details of a limit are far less important to me than the mere benefit of having a limit with its benefits to consistency.

[jonsneyers]

+1 to a tighter and unified limit on the number of bytes sniffed!

[baumanj]

I'm going to look at addressing this via whatwg/mimesniff#143 (basically a PR to add AVIF to the mimesniff algorithm) and some tests on https://github.com/web-platform-tests/wpt. Will report back when done or blocked.