Access to fragment identifiers

[cconcolato]

There are several use cases that could benefit from Service Workers having access to fragment identifiers. Several of my use cases are related to the processing of fragment identifiers for MP4 files. In MP4 files, some identifiers are generic media fragment identifiers like time based (#t=10), and some others are specific to MP4 like video.mp4#track_id=1 or video.mp4#item_name=foo. Native support in browsers for all these identifiers is unlikely to happen. I'd like to polyfill the support for various MP4 fragment identifiers using my Service Worker (see https://github.com/gpac/mp4box-sw).

[annevk]

This is whatwg/fetch#214 I think.

[wanderview]

We should discuss fragments at the f2f next week.

[annevk]

FWIW, I think I favor exposing them. It's a little weird since they don't go over the network, but given the other things we expose to service workers that doesn't seem like a big stretch.

[jakearchibald]

Agreed. The mp4 use-case is particularly compelling.

[wanderview]

I think my main question is how this should work for Cache and Cache.match(). Also redirects.

[annevk]

Cache should probably ignore the fragment in the URL (but still store it). Redirects is a good question. HTML defines some semantics for them (reuse if no fragment on redirect URL, otherwise use redirect URL fragment) which we can possibly reuse here. Haven't thought deeply about the implications though.

[jakearchibald]

F2F: add fragments to request & response urls. Cache API will store them, but will ignore them when matching.

[jakearchibald]

Our intention is that the fragment in a response.url for a navigation will not be used to scroll the page. Only the request url can do that.

[wanderview]

Our intention is that the fragment in a response.url for a navigation will not be used to scroll the page. Only the request url can do that.

To further clarify, I think this is our intention for evt.respondWith(new Response(urlWithFragment)).

I think exposing fragment in response.url would allow scrolling the page if you did evt.respondWith(Response.redirect(urlWithFragment)).

[cconcolato]

Just to make sure I understood. It would be ok for a SW to receive a Request with a fragment identifier and to respond with a Response without (or with a different) fragment identifier, right? For instance, for a URL saying "file.mp4#t=10", if my SW trims the first 10s of the video, I would simply respond with a URL saying "file.mp4" to prevent the browser from seeking 10s further into that Response to t=20.

[annevk]

@cconcolato I don't think that would work. The browser does that seeking on the basis of the request URL.

[cconcolato]

The browser does that seeking on the resource returned by the SW, right? But based on which URL, the original URL in the markup or JS or the URL returned by the SW?

[NekR]

@cconcolato

on the basis of the request URL

You call it original URL.

[wanderview]

@cconcolato You would need to use evt.respondWith(Response.redirect('file.mp4')) to change the URL bar contents and avoid jumping to the location.

[wanderview]

Gecko bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1264178

[annevk]

@wanderview if foo#foo redirects to bar, you end up at bar#foo, so I'm not sure it's that simple.

[annevk]

I'm going to go ahead and assume we plan on doing this for the request URL only.

Where @wanderview talks about redirects above, it's the URL in Location that would have a fragment, not the response URL. I don't think there's a way to get a fragment in a response URL and I don't think we should be adding that.

[wanderview]

Where @wanderview talks about redirects above, it's the URL in Location that would have a fragment, not the response URL. I don't think there's a way to get a fragment in a response URL and I don't think we should be adding that.

This confuses me. If I do fetch(urlThatRedirects) shouldn't the results response.url be the final redirected url? Isn't this step 11 of Http-Network Fetch? It would seem to me you can get a fragment in the response URL that way, no?

[annevk]

Say I request A. A returns a redirect with Location set to B#foo. Then I request B (I keep #foo from the network). Then I get a response for B. Then I scroll to #foo or do nothing.

The key point here is that responses with a URL come from the network, and requests never include fragments over the network. The change here is that we do expose fragments to service workers and from Request objects in general. Similar to how we expose them elsewhere in the platform.

[wanderview]

But we explicitly set the Response object's url list to the Request url list. Where do we remove the fragments when that is done? I don't see this defined anywhere.

[annevk]

Doh. Hmm. I guess I would still prefer not exposing the fragment on the response since it doesn't really make much sense. We mostly copy the URL from the request since there's no other more canonical location to get it from.

[jungkees]

F2F: add fragments to request & response urls. Cache API will store them, but will ignore them when matching.

I'm planning to work on this. This will change cache methods to not remove the fragments (store them as given) and Query Cache algorithm to exclude the fragments when matching the urls. To be clear, scope urls will still be stored without fragments and matched without them.

@annevk Does fetch have any dependencies that will break things if I invoke it with a url having a fragment? Otherwise, I'll update the SW part of it.

[annevk]

I don't think so. Please use https://url.spec.whatwg.org/#concept-url-equals for URL matching.

It's still not entirely clear to me what to do with the case where we set the request URL on the response. I think we basically want to be able to not care about the fragment there.

[mkruisselbrink]

Also for the request in ForeignFetchEvent we probably don't want to expose the fragment, as normally the foreign origin would never have had access to the fragment?

[annevk]

I guess not, although that seems somewhat unfortunate. That makes the same-origin sharing also somewhat weird since same-origin services that make use of it could never become cross-origin services. Hmmmmm.

[jakearchibald]

So to sum up:

• Fragment appears on request.url
• Fragment absent from response.url
• request.url including fragment is retained in the cache
• Fragment ignored by cache.match
• Fragment stripped from registration scope

[jungkees]

I'm confused while working on it. What's the point we store the fragment in the cache if we don't use it to match?

cache.put('foo#1', response1).then(() => { cache.put('foo#2', response2); });

Should it make two entries ['foo#1', response1], ['foo#2', response2] in the cache map? Or should the second put replace the first entry resulting in a single entry ['foo#2', response2]?

cache.matchAll('foo#3') later will return a promise resolves with [response1, response2] or [response2], respectively.

What if we do cache.addAll(['foo#1', 'foo#2', 'foo#3']); and cache.matchAll('foo#4');?

If addAll creates three different entries, the matchAll is expected to return three identical response objects in an array? Or should it aggregate it to a single response object array?

Or the addAll above should create a single entry resulting in ['foo#3', response]?

[jakearchibald]

On the foreign fetch front, there are already cases where the fragment kinda reaches the server, through range requests on media.

[jakearchibald]

I'm confused while working on it. What's the point we store the fragment in the cache if we don't use it to match?

I think it's to ensure the request you get out of the cache is, as much as possible, like the request you put in. I don't think we should use it for matching.

cache.put('foo#1', response1).then(() => cache.put('foo#2', response2));

I expect the cache to contain one entry (response2) after this.

[jungkees]

Thanks! @jakearchibald also pointed that cache.addAll(['foo#1', 'foo#2', 'foo#3']); will reject due to 2.3.2.3 https://slightlyoff.github.io/ServiceWorker/spec/service_worker/#batch-cache-operations-algorithm

[annevk]

On the foreign fetch front, there are already cases where the fragment kinda reaches the server, through range requests on media.

Yeah, since it doesn't affect navigation I'm actually inclined to allow this, since it's unlikely fragments for "subresources" have confidential information in them (typically they're just not present). We should maybe open a distinct issue to discuss that though.

[mkruisselbrink]

On the foreign fetch front, there are already cases where the fragment kinda reaches the server, through range requests on media.

Yeah, since it doesn't affect navigation I'm actually inclined to allow this, since it's unlikely fragments for "subresources" have confidential information in them (typically they're just not present). We should maybe open a distinct issue to discuss that though.

Sounds good to me.

[annevk]

Filed #910 on that. The change in Fetch for requests is tracked by whatwg/fetch#214.

Anything else?

[jungkees]

Addressed the SW part of it. Let me know if you think we missed anything.

[mfalken]

Chrome status: https://www.chromestatus.com/feature/4753419730419712
Chrome bug: https://bugs.chromium.org/p/chromium/issues/detail?id=624278