Etags: solutions besides clearing/disabling cache [which does clear/block etags, sorry about the confusion]

[Thorin-Oakenpants]

snip

[Atavic]

Cookieless Cookie Fake Test: The test page mentioned by Pants uses the IP address for session tracking and the user agent but not ETags.

This test should use ETags, instead.

[Just-me-ghacks]

http://lucb1e.com/rp/cookielesscookies/

Results:
browser.cache.memory.enable TRUE & same IP - I was tracked.
browser.cache.memory.enable FALSE & same IP - I was not tracked.
browser.cache.memory.enable TRUE & different IP - I was not tracked.

This "test" could be fake! We need a more reliable ETag test.

[Atavic]

Noscript. NoScript's ABE (Application Boundaries Enforcer) allows stripping etags.

Secret Agent generates spoof ETags headers with random values, preventing ETags being misused for tracking.

Run a local proxy as Privoxy that can block ETag headers.

[2glops]

Tor Browser 7.0.2 (vanilla)
TB with stored text > tracked by number of visits but not by stored text
TB with stored text + new identity > not tracked

[Atavic]

Installed Modify Header Value (HTTP Headers) on FF ESR 52.0
In HTTP Headers options I created a new entry with:

URL *
Header Name If-None-Match
Header Value *
Then I picked Add.

The test passed.

I get 2 Visits repeatedly if I pick Remove instead of Add.

[Forsaked]

@Atavic This way every other Website breaks its layout/design.

[Atavic]

You can whitelist sites, deactivate your filtering choice or launch a second browser with this WebExtension on AMO, part of these projects by Andy Portmen.

The HTTP Entity Tag is used to reduce the server bandwidth when a resource was previously stored in the cache. Abusing the ETag values allows browser's tracking: the ETag value is unique as an MD5 or SHA1 hash. You can either strip the hash value from every If-None-Match header value with Privoxy or spoof it with Secret Agent Add-on: its spoofing feature adds a random hash to outgoing requests, while its whitelist feature allows you to specify trusted hosts. For Privoxy issues there's an active forum.

[ghost]

LOL!

Hmmm .. where did zymase's comment go? .. Yup, this is testing ETags zymase, there is no JS, no cookies, no dom etc etc

That's why I had removed my comment. Thanks for not including in your brief recall of my shame the fact I love chocolate and mini-skirts!

Yeah, I had commented too quickly, suggesting the LocalStorage to be an explanation of the cache issue detailed by this thread. I then took the time to read extensively, and at this time I've installed The Secret Agent add-on (from developer's page mentioned above by Atavic, when unavailable on AMO).

Hey! I'm a secret agent man as sung by Austin Powers; yeah! -- Seriously, the add-on is most interesting, testing it right now as I said. One thing is sure, the cache issue revealed, exposed, applied with the ETag Test PoC is obsolete with Secret Agent.

[ghost]

After some time testing the Secret Agent add-on I'm finally keeping it but will be using only its eTags & Caching feature because it seems to handle the eTag intrusion correctly by spoofing it.

Secret Agent has several other features but too many sites get problematic when these features are enabled. Of course the add-on has a whitelist which is flawless but I've had to whitelist domains, sites only because of the features I then removed. But whitelisting a site will then stop filtering its eTags as well.

In other words, only eTag spoofing and no sites (up to now) need to be whitelisted

Secret Agent then but only for eTag spoofing, which it does well, and which resolves the very concern of this thread, as I see it.

One last thing: Secret Agent is displayed with a toolbar button on its dedicated toolbar, positioned right under the urlbar and reserving the screen width for only its toolbar button followed by description of the add-on's status (what it's doing, is it on or off). Removing the toolbar button is possible, can then be positioned anywhere and the toolbar itself can be hidden with right-click on the tabs toolbar then uncheck Secret Agent toolbar. Of course Secret Agent toolbar button alone will not be followed by description of status as when it is on its toolbar, but hovering it provides the information and right-clicking it opens a drop-down menu for whitelisting the site, disabling/enabling Stealth Mode and access to Options.

Done this way, seems interesting to me.

[ghost]

I don't like the idea of needing another addon (and why isn't it in AMO anyway? Not that I would use it if it were).

As I and the author understand it, cache in general is the problem: "[...] cache tracking issues [...]" and "[...] I currently have no straightforward answer since cache tracking is virtually undetectable, [...]" .
I don't understand, why is cache not cleared if Firefox is closed?
This tracking seems to be disabled if browser.cache.memory.enable is set to false (#179 (comment)).
I would prefer deleting the cache if Firefox is closed. This seems not to work? (#179 (comment)) What's the status on that?

[ghost]

@Thorin-Oakenpants Thanks for your work then. This is indeed intriguing. I think this must be local indeed, because if I use a second profile, the tracking-cookie isn't there. I wouldn't bother with ETags, because the cache is the generic issue here and who knows that other techniques are used using cache. The real issue is, as you pointed out in your initial post, is that the cache is not being cleared -- if it's set to be cleared -- when FF is closed. I'd personally would want the cache to be cleaned when FF is closed (expected behavior), or I would disable cache altogether (not the preferred solution).

[ghost]

Another fingerprinting test, and this one is the toughest I know : Cross-browser fingerprinting test 2.0

[ghost]

Of course it's always possible to add a uBlockO filter to block Cross-browser fingerprinting test 2.0, such as /evercookie.js$important but that solves a test, an effect and not the cause. Moreover I doubt a site using evercookies would name their script evercookie.js.

[2glops]

@zymase
You're right and I just deleted my comment after a short time while you were answering.
Gulty ! That will not happen again.

[ghost]

@2glops it happens to everyone! If deleting a comment is authorized it's for a good reason.

So, we can use medicine to block evercookie.js (namely) but we cannot cure the fingerprinting disease :)

[crssi]

About ETags and performace...
To get rid of ETags, I have set:
user_pref("browser.cache.memory.enable", false);
user_pref("browser.cache.memory.capacity", 0);

No ETags, but I don't see any performance issues too.

[crssi]

^^Is any of those master switch and I don't need the other one?

[crssi]

As I say, I don't see any performance issues. Why don't you try it?

[crssi]

That is OK and understandable. But I have asked why don't YOU try it. I haven't implied anything further, at least not to include it into public version.

[earthlng]

you can try this one: https://addons.mozilla.org/en-US/firefox/addon/header-editor/
I have my own WE to do these kinds of things but I'm not gonna release it anytime soon. (or maybe ever)
You can't remove If-None-Match though because it's one of the headers that gets added after WEs had their hands on them. But removing the ETag Response Header should work.

[crssi]

For me too...pleaseeee :)

[earthlng]

The code doesn't use .toLowerCase to compare headers but I think existing headers are now always lowercase, meaning you need use lowercase when you want to remove (=use an empty value for it) a certain header. BUT if you want to add one you should probably use the correct casing, fe. if you always want to add a Referer Header, name it 'Referer' not 'referer' or you might make yourself pretty unique because nobody else sends a lowercase referer header!
Anyway, I think this should do:

you can do some testing on http://httpbin.org/ if needed.

[crssi]

Thank you @earthlng, this seems to get rid of etags with memory cache enabled.

Nice

[crssi]

Test at http://lucb1e.com/rp/cookielesscookies/ also passes in desired way.

[earthlng]

If-None-Match sends the Etag if one was received but since you're now removing it there's nothing to send. And you can still profit from caching because of the Last-Modified header. That could also theoretically be used to fingerprint by sending unique dates for each user and request but I don't think anyone does that (yet?!). If you want to remove that too you might as well just disable caching again.

[earthlng]

Shame I used a shit extension to test it.

comments like that are part of why I'll probably never release an extension again. Just because people don't understand how it works and what it does, doesn't make it shit. Just sayin'

[earthlng]

It didn't work because FF adds the If-None-Match header after changes made by WEs. Not knowing that doesn't make you an idiot but ranting about how shit it is makes you kind of an asshole. It even clearly states

Modify Header Value can add, modify or remove an HTTP-request-header for all requests on a desired website or url.

obviously if you try to modify a response header with that extension it won't work.
There's a big difference between "it doesn't work - this is shit" and "I can't get this or that to work - can you help, please", wouldn't you agree?

[earthlng]

Re: last-modified - you could use a custom function to add a random number of seconds or minutes but since I doubt that this FP technique is commonly used, it would make you stand out more on most websites than just sending back the original value.

[crssi]

Sure I am. :)

[claustromaniac]

and therefore isn't really testing ETags specifically

If I interpreted the source correctly, it is. The fact that the PoC doesn't hold when you change your IP or user-agent is a side effect of how the former generates the unique ETag to track you. As long as you don't change your IP or user-agent, you can still use it for the purpose of checking if your browser is sending If-None-Match headers or not.

I would suggest you to make sure nothing in your setup is changing your user-agent. Also, make sure to reload the page to see updated results. From the PoC:

To demonstrate how this works without having to use Javascript, I had to find a piece of information that's relatively unique to you besides this ETag. The image is loaded after the page is loaded, but only the image contains the ETag. How can I display up to date info on the page? Turns out I can't really do that without dynamically updating the page, which requires javascript, which I wanted to avoid to show that it can be done without.

This chicken and egg problem introduces a few bugs:

• All information you see was from your previous pageload. Press F5 to see updated data.
• When you visit a page where you don't have an ETag (like incognito mode), your session will be emptied. Again, this is only visible when you reload the page.

I did not see a simple solution to these issues. Sure some things can be done, but nothing that other websites would use, and I wanted to keep the code as simple and as close to reality as possible.

Note that these bugs normally don't exist when you really want to track someone because then you don't intend to show users that they are being tracked.

[vertigo220]

Not changing my IP or user agent, and I'm refreshing the page over and over trying to figure this out. And I guess I'm more confused than I thought, since I was understanding the issue to be that, since it uses your IP to generate the ETag, if the IP doesn't change it causes the same ETag to be generated again, making it appear as though it was never blocked, when in fact it's just replaced by an identical one.

[claustromaniac]

@Thorin-Oakenpants we're removing the response etag because that prevents the browser from storing the etag, which in turn prevents it from sending the if-none-match header in the future (which can be used to track us). I just re-tested the PoC and had the same results as you.

To avoid further confusion, we should forget about the PoC for a moment. We don't need it to test this.

1. Disable the rule in Header Editor.
2. Load any site that caches an image (https://www.ghacks.net/2017/12/09/a-solution-to-etag-tracking-in-firefox/ for example). Clear the cache for that site first if necessary.
3. Open Dev Tools and reload, check the headers for one of the images. On first load the status will be 200 and you will have Etag in the response headers but no if-none match in the request headers. You won't see this unless you do a clean reload after you open the dev tools, but that's OK. Once the image is cached, you should have status code 304 (not modified) and you should have both etag response header (as before) and if-none-match request header.
4. Enable the rule in Header Editor, clear the cache for the site (with shift+click on the reload button) and re-test. This time you'll see that the second time you reload you will still get a 304 status code for the image (it will be cached) but you won't have any if-none-match in the request headers.

Now, if you ask me why the etag still shows in the response headers (when the Header Editor rule is supposed to be removing it), my guess is that whatever hook Header Editor uses for modifying the response header kicks in after the Dev Tools load that info.

As for what is going on with the PoC... well, I say F it. It probably has to do with the other cache headers or something.

[claustromaniac]

Not sure what to tell you, it sure works for me. You might need to reload a couple of times to get the 304 status code. Other than that, maybe use the proper casing in the Header Editor rule for the etag header? As in ETag instead of etag? I think it shouldn't matter but.. who knows.

Also keep in mind that whatever image you use to test needs to have a decent time to live, otherwise the server will refresh the image in your cache more often and that could mess with this test.

Javascript could mess with that too, make sure it's disabled for that site for the purpose of testing.

[claustromaniac]

Weird. Both ETag and etag seem to work on my end. Sounds like it is unrelated. Maybe it's just one of those weird timing things that probably happen on the server side. Who knows.

Wait a while and try again with etag if you're curious. I don't know lol

[claustromaniac]

are you using the user.js basically as is?

I'm using mostly minimal overrides on this profile. I enabled most of the Personal section, enabled disk cache, and relaxed the shoulder-surfing and related stuff I don't need to worry about.

[vertigo220]

@claustromaniac's method worked for me. Glad that's finally sorted out, been working on it on and off for a few days now. @Thorin-Oakenpants, I had the opposite problem as you at first: the tag wasn't present even after disabling Header Editor and reloading the page again. I wonder if your problem is related but in reverse. Maybe try disabling the rule, shift+F5, F5, enable the rule, shift+F5, F5. That's basically how I got it to show the tag then discard it again, and it seems to be working now. I guess there's some interference somewhere in there with all the back and forth and reloading and cache clearing.

[claustromaniac]

I'm suspecting it might be an issue with the extension. Two of my oldest rules disappeared the last time I restarted the browser (just because).

Maybe it doesn't have anything to do with the casing, but rather with the way the extension saves the settings.. maybe something relevant changed in one of the updates in the time frame between the moment you created that rule and today (that you edited it, or created a new one).

[vertigo220]

I created three rules: etag, ETag, and eTaG. Only the first worked. It also worked with all of them enabled, so maybe a bunch of different rules (ETAG, etag, ETag, Etag, ...) could be made, or maybe it could be done with regex (something like [Ee|Tt|Aa|Gg]). Or would that possibly make it stand out and be more unique? Also, are you testing with the ghacks link from claustromaniac's post? I just assumed you were, but if not that could explain the difference.

[vertigo220]

Not a typo. The first one ("etag") is the one I was using all along, but couldn't get it working (or verify that it was working). What I was saying worked from claustro's post was verifying it with the ghacks site. I'm not sure if the difference was just using a normal site vs the "flawed" test/PoC, or if the time I tried following Kane's instructions it just happened to not work. So does that mean you didn't try with the ghacks site? If that's the case, try that one with "etag."

[earthlng]

OMG! before you guys all go nuts I'll step in here :)

The problem looks like it's caused by a recent change in FF, not the extension. Not too long ago all headers were exposed by FF in exactly the casing sent by the server. Then they changed it to all lowercase for a while and now they apparently changed it back to original casing (IDK when exactly).
@vertigo220 what version of FF are you using?

But the extension code has also changed since the last time I looked at it, and frankly is still less than ideal. For example certain headers can exist multiple times and the extension only mods the 1st one.
For ETag it probably shouldn't matter but you never know. fe. IDK what FF does when a server sends 2 ETag headers.

The best solution that doesn't rely on FF keeping its current way of providing the headers nor any potential future changes in the extension itself, is to use a custom function. That should also work with all FF versions (ESR52.x included) and all current, former and future releases of the extension.
And it bypasses the less than ideal implementation of the extension.

So, change your rule from Normal to Custom Function and paste this:

for (let a in val) {
    if (val[a].name.toLowerCase() === 'etag') { val[a].value = ''; }
}

Don't use this in Chrome though because Chrome preserves empty headers and it might end up sending an empty If-None-Match header, IDK.

You'll know that it works when "Number of visits" never goes higher than 2 @ https://lucb1e.com/rp/cookielesscookies/

[earthlng]

Oh, and I forgot to mention that you can't rely on the devtools for headers because it doesn't accurately show changes made by extensions. In this case you'll see that it lists ETag in the response header despite it having been removed but you can deduce that the removal worked when you refresh and the next request header doesn't include If-None-Match. So despite being less than ideal the devtools are still somewhat useful because of the connection between these 2 headers.

[vertigo220]

Thanks @earthlng! That function is the only way, besides Secret Agent, that I've been able to pass the test at that site. And it's good to know it's more robust. Am I correct in understanding that code doesn't change the case, so etag=xyz will be set to etag="", ETag=xyz will be set to ETag="", and so on?

Also, I'm using Waterfox 56.2.0x64.

An OT question: could you or someone else tell me how I would make the following redirect work:

^http(s?)://www.amazon.com/gp/buy/spc/(.*) > https://smile.amazon.com/gp/buy/spc/handlers/display.html?

I figured out how to do it with Request Control, but since it seems Header Editor is capable of much more, I'm trying to transfer the rules from RC into it so I can reduce my number of addons. I've tried different variations, and the one I pasted here worked once (I think), but hasn't worked since.

[earthlng]

Am I correct in understanding that code doesn't change the case, so etag=xyz will be set to etag="", ETag=xyz will be set to ETag="", and so on?

yes it doesn't change the case.

I'd recommend to stick with RC or Redirector for redirects because it's a lot easier to match precisely the request types that you want/need.

[vertigo220]

Thanks. I had my doubts, since it doesn't follow the "rules" in the examples from the wiki, but it does work, sort of. Unfortunately, it doesn't work when landing on the page, only when refreshing it, so I guess I'll have to ask on the HE repo.

[earthlng]

HE is not great for redirects because I think it matches all request types. That can be pretty terrible actually.

Unfortunately, it doesn't work when landing on the page, only when refreshing it

It's probably not a normal link but uses some JS to load it. Use RC with request type Document and then always middle-click those links to open in a new tab, that should work.

[claustromaniac]

OMG! before you guys all go nuts I'll step in here :)

If that includes me, I'm sorry to say you're too late. Like...years late.

Thanks for the explanation and the solution, mate! BTW did someone translate the HE documentation to english without me finding out? Last time I checked it was only in japanese IIRC. Do you know japanese? 😝 (EDIT: NVM. I see that part is translated now)

I forgot to mention that you can't rely on the devtools for headers because it doesn't accurately show changes made by extensions. In this case you'll see that it lists ETag in the response header despite it having been removed but you can deduce that the removal worked when you refresh and the next request header doesn't include If-None-Match.

I pointed this out in my post above and everyone seemed to understand it. If my guess is right the inaccuracies should be in the response headers, especifically. Did you ever happen to investigate that too, by any chance? Just curious.

[claustromaniac]

BTW, one of my HE rules that disappeared from my list today just reappeared a while back. LOL! The other one is still MIA.

[claustromaniac]

Clastro - have you looked inside the HE storage?

And what should I be looking for in there, exactly? I'll probably wipe it and rewrite the few rules that I use, anyway. It's not much of a hassle.

I'm still wondering why we got different results with the casing in our tests earlier. I only had issues with the lowercase etag rule when I tried the PoC, but both etag and ETag seemed to work for me everywhere else.. Then again, I didn't test much. I didn't stay around here for too long.

[claustromaniac]

I'm not using groups either. I just had 5 rules and nothing really complicated. I wouldn't mind digging into the issue in other circumstances - and I probably will, if it turns out to be a recurring issue - but I'm pretty sure I'll invest less time rewriting them for the time being.

Plus, I get to save you the hassle of helping me with that! How nice is that? You sound tired enough as it is, 👖

[vertigo220]

@earthlng said:

It's probably not a normal link but uses some JS to load it. Use RC with request type Document and then always middle-click those links to open in a new tab, that should work.

It is JS, which is what gave me a hard time originally, but I already have it working in RC. I was just hoping HE could do the same thing so I could eliminate RC (nothing against it, just wanted to slim things down a bit).

@claustromaniac - Once you rebuild them, you should be sure to export them in case it happens again.

[earthlng]

@claustromaniac

I pointed this out in my post above and everyone seemed to understand it.

yes I saw that, just wanted to re-confirm. And I'm not so sure everyone seemed to understand it :)

If my guess is right the inaccuracies should be in the response headers, especifically. Did you ever happen to investigate that too, by any chance? Just curious.

No I did not. As far as I can tell the inaccuracies are all over the place

[claustromaniac]

I'm not so sure everyone seemed to understand it :)

That's ok. I admit I often come short when I try to explain stuff to others anyway.

No I did not. As far as I can tell the inaccuracies are all over the place

I've never seen inaccuracies in the request headers but I'll try to keep that in mind in case I ever do. I better stop trusting the devtools on that. Thanks again.

[vertigo220]

BTW, here's another, unrelated cache tracking technique you may or may not be aware of which, AFAICT, can't be defeated except by periodically clearing the cache. Thought you might find it interesting.

[claustromaniac]

posted in #436

[claustromaniac]

Done.