Fix transitions with non-recommended headers

[Fryuni]

Changes

The syntax of the Content-Type header specified by the HTTP Semantics RFC is as follows:

Content-Type = media-type

OWS = *( SP / HTAB )

media-type = type "/" subtype parameters

subtype = token
type = token

parameter = parameter-name "=" parameter-value
parameter-name = token
parameter-value = ( token / quoted-string )
parameters = *( OWS ";" OWS [ parameter ] )

tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
 "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA
token = 1*tchar

The ; that marks the start of the parameters may, even though not recommended, be preceded by any number of white spaces. The previous implementation of the fetchHTML function would not accept a header in that form and do full page reload.

This PR takes that into account and replaces the media type extraction from the header with an implementation that works for any spec-compliant server, even without using recommended practices.

Testing

A new integration test was added for a page replying with the following header:

Content-Type: text/html ; charset=utf-8

Docs

I don't think this needs any docs change

[changeset-bot]

🦋 Changeset detected

Latest commit: 83134ec

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[martrapp]

I read about it in your article ;-)
Many thanks for this improvement!