Spoof Referer header of third party requests #2155

troy-lamerton · 2018-09-02T18:37:42Z

This PR will spoof the Referer header of all third party GET requests.

For other third party requests, the Referer header is removed - this is the same as what uMatrix does.
See this comment in uMatrix and the implementation in uMatrix

Fixes #855 except that spoofing is mandatory, there is no option to turn it off.

Should an option be added for this? Is it not needed?

bcyphers · 2018-09-03T17:57:12Z

This would be a nice feature in most cases, but I think it's known to cause issues with some sites/widgets (see gorhill/uMatrix#68). We should definitely have a toggle for this, and perhaps a per-domain toggle as well.

ghostwords · 2018-09-04T22:51:57Z

Relevant Firefox breakage study: https://blog.mozilla.org/data/2018/01/26/improving-privacy-without-breaking-the-web/

bcyphers · 2018-09-06T22:38:30Z

src/_locales/en_US/messages.json

@@ -135,6 +135,10 @@
        "message": "Replace social widgets",
        "description": "Checkbox label on the general settings page"
    },
+    "options_spoof_referrer_checkbox": {
+        "message": "Spoof the Referrer header",


It would be useful to have this text link out to an explanation of what a "referer" header is. Also, should we spell it the English way, or the canonical way? https://en.wikipedia.org/wiki/HTTP_referer

bcyphers

Couple very minor nits. Looks good otherwise.

bcyphers · 2018-09-07T00:48:52Z

src/js/webrequest.js

+  } else if (badger.isSpoofReferrerEnabled()) {
+    if (badger.isPrivacyBadgerEnabled(tabDomain) && badger.isPrivacyBadgerEnabled(requestDomain)) {
+      // spoof referer header for third party requests
+      const refererHeader = details.requestHeaders.find(header => header.name === "Referer");


Should we do a case-insensitive check here? header.name.toLowerCase() == "referer"

popcornenthusiast · 2018-09-08T02:38:35Z

I know for a fact that when the referrer header is spoofed/wiped out for third-parties, embedded private Vimeo videos which can only be embedded on certain site/domains, can't be reproduced, as Vimeo gets the site/domain from the referrer header. More info here and here. So, on this matter, I would go with what I said some days ago:

Wipe out the "referrer" header from connections to third-party cookie-blocked domains AND blocked domains (though, I'm guessing, this would be redundant);

With "third-party cookie-blocked domains" being the ones already blocked by PB.

bcyphers · 2018-09-08T02:53:05Z

@popcornenthusiast interesting, thanks for bringing that up. Maybe we should hold off on this for a bit.

We do already remove the referer from cookieblocked domains: https://github.com/EFForg/privacybadger/blob/master/src/js/webrequest.js#L229

popcornenthusiast · 2018-09-08T20:31:40Z

@bcyphers Oh, you're welcome! And...

We do already remove the referer from cookieblocked domains: https://github.com/EFForg/privacybadger/blob/master/src/js/webrequest.js#L229

Didn't know that. That's great!

But, see, instead of spoofing/wiping out the referrer header from every third-party request, why not wipe it out only from every cookie-blocked domain (already done, like you said) AND from requests for css and fonts from third-parties? It probably wouldn't break anything, this way. Also, is it possible to wipe it out from preconnect/prefetch look-a-like requests?

troy-lamerton · 2018-09-09T17:59:05Z

I don't mind these change being made to this PR, or if someone wants to work from this that's fine to. I don't have the time to work on this anymore.

ghostwords · 2018-09-11T15:23:10Z

Vimeo not working when cookieblocked is tracked in #188.

bcyphers · 2018-09-12T21:13:15Z

@popcornenthusiast we could use the resource type of the request to decide whether to send referer. It looks like Vimeo embeds are loaded as the sub_frame type (at least here). Maybe we can start by allowing referrers to all sub_frame requests?

ghostwords

Thank you!

Left code logic feedback. Still need to consider the overall feature some more.

To recap ... This enforces origin-only Referer values for third-party GET requests, and strips Referer values entirely for third-party PUT requests. Is this right? What's the rationale behind stripping Referer from PUTs?

Do we have more examples of what this feature might break? Do any privacy tools besides uMatrix provide similar functionality?

Edit: Brave does:

Referrer values are modified in the following ways

cross-origin requests for iframes and sub-resources have their referrer set to be the origin of the requested resource

cross-origin navigations have no referrer at all

Brave also keeps a list of referrer exceptions. Vimeo appears exempt, for instance.

Edit2: Brave's wiki seems out-of-date: brave/brave-browser#3422

ghostwords · 2018-09-12T22:23:49Z

src/js/webrequest.js

@@ -172,6 +172,23 @@ function onBeforeSendHeaders(details) {
    } else {
      return {};
    }
+
+  } else if (badger.isSpoofReferrerEnabled()) {


This shouldn't be an "else if", just an "if", since the "if this isn't a third-party request" block above always exits.

We should move this entire referrer spoofing block down in onBeforeSendHeaders callback.

We don't want to do this when Badger is disabled, and we already have a check for this later on in the callback (if (!badger.isPrivacyBadgerEnabled(tabDomain)) ...). We don't want to do this when Badger "cookieblocks" a request (Refer gets stripped there already), or when Badger ends up blocking the request (when the request happens to register as third strike thanks to cookie headers).

We do want this to work in concert with DNT header injection though, so it makes sense for things that alter headers to live next to each other.

ghostwords · 2018-09-12T22:25:40Z

src/js/webrequest.js

@@ -172,6 +172,23 @@ function onBeforeSendHeaders(details) {
    } else {
      return {};
    }
+
+  } else if (badger.isSpoofReferrerEnabled()) {
+    if (badger.isPrivacyBadgerEnabled(tabDomain) && badger.isPrivacyBadgerEnabled(requestDomain)) {


We shouldn't check requestDomain here, right? Privacy Badger's site whitelist is for first-party (site) domains.

ghostwords · 2018-09-12T22:34:03Z

src/js/webrequest.js

+        if (details.method === "GET") {
+          // spoof referer value
+          const requestUrl = new URL(details.url);
+          refererHeader.value = requestUrl.origin;


I think we already have the origin in the requestDomain variable defined above.

ghostwords · 2018-09-12T22:35:17Z

src/js/webrequest.js

+  } else if (badger.isSpoofReferrerEnabled()) {
+    if (badger.isPrivacyBadgerEnabled(tabDomain) && badger.isPrivacyBadgerEnabled(requestDomain)) {
+      // spoof referer header for third party requests
+      const refererHeader = details.requestHeaders.find(header => header.name === "Referer");


Style nit: refer_header (lowercase with underscores for for non-object/non-object-property variables).

ghostwords · 2018-09-12T22:35:35Z

src/js/webrequest.js

+          refererHeader.value = requestUrl.origin;
+        } else {
+          // remove referer header from non-GET request
+          const refererHeaderIndex = details.requestHeaders.indexOf(refererHeader);


Style nit: referer_header_index

popcornenthusiast · 2018-09-13T18:17:18Z

@bcyphers

we could use the resource type of the request to decide whether to send referer.

That sounds good! Then, instead of not sending the referrer only to css and fonts requests, it would be possible to also don't send it to beacon, ping and speculative requests without breaking anything (I'm still not sure about image, imageset and object). Also, does "speculative" takes care of dns-prefetch requests?

It looks like Vimeo embeds are loaded as the sub_frame type (at least here). Maybe we can start by allowing referrers to all sub_frame requests?

Not sure about this, though. Would not it allow tracking from every iframe embedded objects? Vimeo wouldn't get broken anymore, but then the problem would be with tracking.

bcyphers · 2018-09-17T20:05:08Z

@popcornenthusiast What I'm proposing would only allow referrers for <iframe> elements from domains on the yellowlist. I think the potential privacy loss here is very limited, since we're already allowing those domains to make requests without checking the parameters.

But bringing the conversation back to this pull request: since Vimeo is already on the yellowlist, it seems like users are already suffering the negative consequences of Privacy Badger blocking referers, at least in that case. We can fix that issue separately, but I don't think it should affect how we deal with this PR. I'm in favor of merging this in, then dealing with the broken-site issues as they come in. Based on the Mozilla report, I don't think there should be very many.

popcornenthusiast · 2018-09-22T05:21:04Z

@bcyphers Yes, you're right. When I made my first comment I didn't know that Vimeo (and any other service that works the same way) was already broken, and then I just got carried way.

ghostwords · 2018-11-28T18:16:52Z

Besides breaking some Vimeo videos, removing the referrer header also breaks some (?) akamaihd.net videos (#188 (comment)) and some (?) Google map widgets. For example, map tiles won't load on Zillow if www.googleapis.com was set to "yellow".

ghostwords · 2019-11-11T20:28:31Z

Closing as per #2155 (comment)

troy-lamerton added 2 commits September 2, 2018 20:31

spoof referer header of third party requests

d79bec6

remove logs that were for debugging

cced561

Add checkbox to toggle Spoof referrer option

df0a7bb

update spoof setting on checkbox click

2e66ed4

bcyphers self-requested a review September 5, 2018 22:52

bcyphers reviewed Sep 6, 2018

View reviewed changes

bcyphers suggested changes Sep 7, 2018

View reviewed changes

ghostwords requested changes Sep 12, 2018

View reviewed changes

ghostwords force-pushed the master branch from 5afce52 to 257be9c Compare September 18, 2018 15:19

ghostwords closed this Nov 11, 2019

ghostwords mentioned this pull request Apr 17, 2021

Could Privacy Badger integrate some features of Privacy Possum? #2761

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spoof Referer header of third party requests #2155

Spoof Referer header of third party requests #2155

troy-lamerton commented Sep 2, 2018

bcyphers commented Sep 3, 2018

ghostwords commented Sep 4, 2018 •

edited

Loading

bcyphers Sep 6, 2018 •

edited

Loading

bcyphers left a comment

bcyphers Sep 7, 2018

popcornenthusiast commented Sep 8, 2018

bcyphers commented Sep 8, 2018 •

edited

Loading

popcornenthusiast commented Sep 8, 2018

troy-lamerton commented Sep 9, 2018

ghostwords commented Sep 11, 2018

bcyphers commented Sep 12, 2018 •

edited

Loading

ghostwords left a comment •

edited

Loading

ghostwords Sep 12, 2018

ghostwords Sep 12, 2018 •

edited

Loading

ghostwords Sep 12, 2018

ghostwords Sep 12, 2018 •

edited

Loading

ghostwords Sep 12, 2018

ghostwords Sep 12, 2018 •

edited

Loading

popcornenthusiast commented Sep 13, 2018

bcyphers commented Sep 17, 2018

popcornenthusiast commented Sep 22, 2018 •

edited

Loading

ghostwords commented Nov 28, 2018

ghostwords commented Nov 11, 2019

Spoof Referer header of third party requests #2155

Spoof Referer header of third party requests #2155

Conversation

troy-lamerton commented Sep 2, 2018

bcyphers commented Sep 3, 2018

ghostwords commented Sep 4, 2018 • edited Loading

bcyphers Sep 6, 2018 • edited Loading

Choose a reason for hiding this comment

bcyphers left a comment

Choose a reason for hiding this comment

bcyphers Sep 7, 2018

Choose a reason for hiding this comment

popcornenthusiast commented Sep 8, 2018

bcyphers commented Sep 8, 2018 • edited Loading

popcornenthusiast commented Sep 8, 2018

troy-lamerton commented Sep 9, 2018

ghostwords commented Sep 11, 2018

bcyphers commented Sep 12, 2018 • edited Loading

ghostwords left a comment • edited Loading

Choose a reason for hiding this comment

ghostwords Sep 12, 2018

Choose a reason for hiding this comment

ghostwords Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

ghostwords Sep 12, 2018

Choose a reason for hiding this comment

ghostwords Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

ghostwords Sep 12, 2018

Choose a reason for hiding this comment

ghostwords Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

popcornenthusiast commented Sep 13, 2018

bcyphers commented Sep 17, 2018

popcornenthusiast commented Sep 22, 2018 • edited Loading

ghostwords commented Nov 28, 2018

ghostwords commented Nov 11, 2019

ghostwords commented Sep 4, 2018 •

edited

Loading

bcyphers Sep 6, 2018 •

edited

Loading

bcyphers commented Sep 8, 2018 •

edited

Loading

bcyphers commented Sep 12, 2018 •

edited

Loading

ghostwords left a comment •

edited

Loading

ghostwords Sep 12, 2018 •

edited

Loading

ghostwords Sep 12, 2018 •

edited

Loading

ghostwords Sep 12, 2018 •

edited

Loading

popcornenthusiast commented Sep 22, 2018 •

edited

Loading