Disconnect.me filter lists

[elad]

Hey,

The recently published Tracking Protection in Firefox For Privacy and Performance reports blocking tracking improves browsing speed, especially when browsing popular (news) sites. They use filters from Disconnect and since I couldn't find them in the settings, I figured I'd make a note about it. Here are the filters I managed to find, all under GPLv3:

• https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
• https://s3.amazonaws.com/lists.disconnect.me/simple_malware.txt
• https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
• https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt

[gorhill]

You're are already better covered with uBlock's default settings: uBlock and others: Blocking ads, trackers, malwares.

[d4k0]

You're are already better covered with uBlock's default settings: uBlock and others: Blocking ads, trackers, malwares.

I'm using this Firefox feature together with uBlock (I also activated several optional filters) since a few weeks and on some sites (I especially noticed this on news sites) Firefox tells me that it blocked some content, so the Disconnect filters still block a few items that aren't covered by the other filters.

[gorhill]

Firefox tells me that it blocked some content

Do you know for a fact that Firefox blocks the content after uBlock? If so, do you have a reference for this?

[d4k0]

From the document that @elad linked:

When the browser is ready to make a network request, Firefox suspends it until Tracking Protection determines whether the destination URL is a tracking domain. If so, Firefox cancels the request before any data is sent or received. Canceling HTTP requests leaves the respective content loaders for documents, images, and scripts in a failed state. The network layer notifies individual loaders of cancellations and emits a document-wide security event.

It seems to block the content before uBlock (sorry if my previous post gave the impression that it would block items after uBlock) which makes sense since this feature is implemented directly in Firefox (I think I also read somewhere that it works like this).

[gitarra]

Firefox Tracking Protection is executed after uBlock (for me at least).

This can be easily observed by going to http://adguard.com/en/filterrules.html

Ublock enabled: ublock blocks googleads.g.doubleclick.net and mc.yandex.ru

Disable ublock: Firefox Tracking Protection icon shows up and Web Console shows Tracking Protection blocking googleads.g.doubleclick.net and mc.yandex.ru

Edit: FWIW I have also noticed Firefox Tracking Protection does cover certain things that uBlock-included rulesets dont. E.g. tracking protection kills some Google Plus integration/tracking on third-parties like youtube with ublock enabled.

[gorhill]

I imported all of the four lists as custom filter lists, and my number of network filters went from 52,833 to 63,733, so 10,900 more network filters. I should have tried that first, did not think about it.

However, simple_ad.txt seems to be almost all covered by current default filter lists, only 4 entries were used out of the 2,727 entries.

The 3 other filter lists do seem useful though:

[gitarra]

Even after adding those 4 filter lists to uBlock, Firefox Tracking Protection kicks in on YouTube.

Seems like it does some other magic as well than just blocking using those 4 domain lists.

[gorhill]

@gitarra it could be just a reporting quirk, this would need to be ruled out before concluding anything.

[Betsy25]

@gorhill, I added those 4 custom lists, however a browser restart later, although they all still are in the editbox, only the first of them is listed under the "Custom" header. (Before the restart, it showed them all having the exact number of used filters as your screenshot)

FYI, This happened with the dev version before you implemented them by default (47e7405)
Firefox nightly 41, v0.9.7.6

[gorhill]

It's because after confirming that they are indeed not overlapping as I first thought, they are now in the list of stock filter lists. Check under "Malware domains" and "Privacy".

[Betsy25]

Sure, I see now, this problem is, I was using the dev version where you didn't implement them by default yet. (The version with this as the last commit : 93a3965)

[gorhill]

They still should appear in there, the list of filter lists is downloaded when a new version become available.

[Betsy25]

EDIT: Doh! Sorry for the confusion.

[gorhill]

Screenshot

They are there, and selected: "Malvertising filter list by Disconnect‎" etc.

[gitarra]

@gorhill Doesnt seem like a reporting quirk. The same request shows up in uBlock request log as allowed (no filter) and blocked by Firefox tracking protection in Web Console. Also Firefox tracking protection blocks widgets.outbrain.com/outbrain.js on cnn.com which isnt included in any of those four lists linked to above and is whitelisted in uBlock.

Looks like its safe to say uBlock cannot completely replace the built-in tracking protection in Firefox.

By the way heres the related Firefox bugs:

(tracking-protection) Provide Usable and Effective Third-Party Web Tracking Countermeasures (Meta):
https://bugzilla.mozilla.org/show_bug.cgi?id=818337

tracking bug for tracking protection:
https://bugzilla.mozilla.org/show_bug.cgi?id=1029886

[gorhill]

Firefox tracking protection blocks widgets.outbrain.com/outbrain.js on cnn.com which isnt included in any of those four lists linked to above and is whitelisted in uBlock

Disconnect used to block outbrain.com: uBlock vs Disconnect diff of filtering results (in red what Disconnect blocked and uBlock did not block, in green what uBlock blocked and Disconnect did not block.)

[gituporabnik]

I have a few questions.

1. Are these links (4 filter lists) that @elad provided all the Disconnect blocklists?
   Are they original and therefore updated frequently?

2. If Firefox Tracking Protection provides a complete alternative to these filter list links, isn't it better to enable only the Firefox Tracking Protection (without adding these lists to uBlock)? It could result in a better performance due to direct Firefox implementation (as @d4k0 stated).
   I just figured out that Facebook has recently enabled the ability to embed Facebook videos on other websites. Firefox Tracking Protection eliminates these videos. In order to retain control it is better not to enable Firefox Tracking Protection.
   To load and show these kinds of videos properly, it is better to add the above 4 Disconnect blocklists (by @elad) in uBlock.

All the exceptions for the websites which implement these Facebook videos can be added using this filter (I found and added 2 websites -besides Facebook- as an example):
||facebook.*$domain=~facebook.com|~24ur.com|~planet.si

EDIT: In case we are using enabled Firefox Tracking Protection instead of uBlock with Disconnect blocklists, we will also have to manually disable each exception website within Firefox Tracking Protection (in addition to the above-mentioned filter).

[Snapy]

@elad Thanks ! I was looking for those lists.
@gorhill Thanks for adding the useful Diconnect's lists to the available filter lists.
@Lencc 1) I think Firefox Tracking Protection uses more than these 4 lists, see the previous comments by gorhill. Yes, they are provided by Disconnect.me so they should be updated frequently.
2) Firefox Tracking Protection isn't a complete alternative but is already part of Firefox. Whether you chose it or an extension with filter lists, the loading time will be improved (see this gHacks article and gorhill's benchmarks)

[gituporabnik]

@Snapy Thank you! I hope the remaining Disconnect blocklists will be added in the future.

I disabled Firefox Tracking Protection and now I only use uBlock to keep control over websites (because of Facebook embedded videos etc.).

[Snapy]

@Lencc You can disable Firefox Tracking Protection per site but uBlock is more flexible especially with the upcoming Dynamic URL filtering.

[d4k0]

1. I think Firefox Tracking Protection uses more than these 4 lists, see the previous comments by gorhill. Yes, they are provided by Disconnect.me so they should be updated frequently.

From the document that @elad posted:

We use a subset of approximately 1500 domains from Disconnect’s privacy-oriented blocklist to identify these unsafe origins [10]. We update the blocklist every 45 minutes to minimize the effects of incorrect blocklist entries.

So they don't seem to use everything.

EDIT:

I just found the following in the comments of the gHacks article:

Similar but not indentical, there is an exception list:
https://github.com/mozilla-services/shavar-list-exceptions

[gituporabnik]

@Snapy You are right, I completely forgot about that. I tested it and it works for the whole domain name (not just for the specific URL at which Firefox Tracking Protection was disabled), which makes it useful enough for me. I guess I am enabling the Firefox Tracking Protection again after all.

I'm sorry for a confusion.

BTW, I also found out that Facebook third-party requests can be blocked all the time - irrespective of embedded Facebook videos, because they don't affect them. Therefore the following custom filters can be added to uBlock (to block the requests in cases when Firefox Tracking Protection / Disconnect filter lists are disabled), which I found on the internet:

||facebook.*$domain=~facebook.com,third-party

||fbcdn.*$domain=~facebook.com,third-party

[lewisje]

I went ahead and added all four of those lists as custom subscriptions, even though it looks like the next release of uBlock Origin will have them built in.

[elad]

Thanks for the fast response @gorhill, but I understand there are two extensions that should be updated. :) @chrisaljoudi?

As for questions about what else is Firefox doing: They're using Google's Safe Browsing API to download filter lists from Google, but those are very likely limited to malware/phishing prone domains, so they add a subset (see previous note about whitelisting) of Disconnect's filters as well. Assuming all of Disconnect's filters are in uBlock, it's unclear to my why there would be any difference with sites like Youtube. (@gitarra, is there still?)

[gitarra]

Yes, as I said Firefox Tracking Protection blocks a request to plus.google.com on youtube videos. Also outbrain is blocked on CNN as I said above.

Theres definitely something else going on than just blocking based on those 4 domain lists. And I doubt google would block their own tracking on youtube ;)

What @gorhill said about disconnect and outbrain would also indicate that using these 4 domain lists isnt the only method of filtering done by disconnect.

Of course you can easily test this yourself, just enable firefox tracking protection.

[gorhill]

using these 4 domain lists isnt the only method of filtering done by disconnect

This is the data used by the Disconnect extension -- outbrain.com is in there:

https://github.com/disconnectme/disconnect/blob/master/firefox/content/disconnect.safariextension/opera/chrome/data/services.json

Note that the Disconnect extension changed ownership it appears and that repo above is no longer maintained it seems.

[gitarra]

Interesting. So their actual blocklists and blocking methods arent public anymore and we dont know what purpose these 4 public "simple" lists serve.

[elad]

I switched to uBlock Origin.