Change default adblock filter lists

[celenityy]

Following up from #1237, I'll give my thoughts regarding filter lists we should consider adding or enabling by default:

To start, I think we should keep EasyList & EasyPrivacy enabled by default. These are both very high quality & well respected lists, & great at blocking undesired content. They also make nice use of the ABP syntax.

So first, I think it'd be a great idea to add HaGeZi's Threat Intelligence Feeds. This is a very comprehensive & effective list at blocking malicious domains, and it would help to make up for the lack of Safe Browsing. Similarly, he also has a list for Malicious IPs, which I also think would be worth including.

Some of HaGeZi's other lists could be considered as well, such as his Dynamic DNS blocking & Badware Hoster blocking. I think his Multi ULTIMATE or Multi Pro++ lists could also make nice additions.

Another list I'd like to suggest considering is my BadBlock project, specifically my BadBlock+ list. This is a very carefully curated & high quality list that goes above and beyond most others. Every domain added is carefully researched & considered, as to avoid false positives. I also make nice use of wildcards & the ABP Syntax. Overall, it's very comprehensive, and I'm putting a lot of effort into growing & expanding it as much as possible, and I think it'd make another very nice addition to Cromite's filters.

I think it'd also be worth including the Divested Combined Blocklist. This is a list maintained by Divested Computing Group (behind ex. DivestOS, Mull, Mulch, Hypatia, etc). and includes a variety of high quality sources, and has nice coverage that most other lists seem to miss. They also have a Fingerprinting blocklist that I think would also make a nice addition if its compatible.

Another list that could be considered is the Actually Legitimate URL Shortener Tool. It is very effective at removing tracking parameters from URLs. My only concern would be that I'm unsure if it's compatible with Cromite's ABP or not, would require testing. I think it could also be worth adding Dandelion Sprout's Anti-Malware List, for similar reasons to HaGeZi's Threat Intelligence Feeds.

I think it'd also be worth including EasyList Cookie. This is great at blocking obnoxious cookie banners and improving privacy.

Additionally, while I'm not sure if it should be enabled by default, I think it'd also be worth at least including Yokoffing's 'Block third party fonts' list. This blocks unnecessary fonts used for tracking on webpages, while also unbreaking pages that legitimately need them.

Another list we could consider adding but not enabling by default is 1Hosts Pro. This is another nice and comprehensive list that I personally use, but it does tend to cause breakage from time to time, so that's why I'd be reluctant enabling it by default.

I think that this is a nice starting point for a discussion, and I'd be curious to hear your thoughts as well as what anyone else has to say or suggest.

[uazo]

thank you for the detailed list.
I anticipate that it's not my priority right now, also because I don't know the world of adblock lists and so I have to document myself first.

However, I anticipate the goals I would like to set for cromite:

• defining default active lists
• check whether it is possible to check lists automatically

regarding the last point is of fundamental importance, because the blocking made by the lists must not be at the expense of privacy or security: if you deactivate functions or scripts, they become nastier, like those of Google, who first try to display ads trying to preserve privacy and if they fail to do so, they don't give a damn and in order to show them, they exploit mechanisms that lead to sending private cookies from the site.

additionally bypassing adblocks is easy and countermeasure methods ineffective or temporary. I can tell you from experience, I have tried it with #251

give me time to consider how to proceed.

[celenityy]

check whether it is possible to check lists automatically

💯, I think this would be ideal. I wonder how other content blockers ex. uBlock Origin & AdGuard handle this with their built-in lists? I might try to research this myself. You're right, I think we need some kind of mechanism in place to protect against malicious or questionable rules.

regarding the last point is of fundamental importance, because the blocking made by the lists must not be at the expense of privacy or security

Yes, 100%. I would much rather a more private & secure browser over slightly better ad blocking, I think most would agree.

if you deactivate functions or scripts, they become nastier, like those of Google, who first try to display ads trying to preserve privacy and if they fail to do so, they don't give a damn and in order to show them, they exploit mechanisms that lead to sending private cookies from the site.

Do you have any sources or more info on this? I wouldn't mind investigating this as a list maintainer myself to figure out the best way to handle such cases. What you say doesn't surprise me though.

additionally bypassing adblocks is easy and countermeasure methods ineffective or temporary. I can tell you from experience, I have tried it with #251

Yeah, content blocking in general is a cat and mouse game :/. I just read through the issue, and I think it might be worth investigating if we could add the uBlock filters as well, since they're especially effective at this. I didn't mention them above though because I'm not sure how compatible they are with ABP, so that will need more investigation.

give me time to consider how to proceed.

👍 Best to do this the right way rather than rushing into it, I agree that we should always put privacy & security first. If there's anything that I can do to help from my end as a list maintainer, please let me know.

[uazo]

Do you have any sources or more info on this?

personally tried, no, I don't think there is anything on the Internet.
Just reactivate this patch (from log to actual block) and the ad scripts become bad!
obviously those not blocked but only hidden (and there are many).

I just read through the issue, and I think it might be worth investigating if we could add the uBlock filters as well

I will get back to that issue and let you know.
from what I had seen, there is a lack of real adblock support from v8, such as blocking not the whole script, but only parts of it.
I stopped on the analysis because even if I introduced something about it, it would then be specific to cromite and I would not have time to create specific rules.

If there's anything that I can do to help from my end as a list maintainer, please let me know.

be sure of it!

[wildbeetle]

Oh no, please don't add those blocklists or enable them by default.

I once experienced a case where websites/domains could not be accessed after trying to add Hagezi Threat Intelligence Feeds and others (forgot which ones) to my OpenWRT router.
And in my experience, hagezi "Multi LIGHT , Multi NORMAL and Multi PRO" are those that do not cause problems.

You need an admin if you want to use aggressive blocklists and that's not good for end users.
You just need “set it and forget it filters” for end user.

I don't have many tried filter lists other than oisd and hagezi.
So can't comment on the others.

Also, those lists can simply be handled/managed by app such as rethinkdns, personaldnsfilter, invizible pro and others for system-wide not just the browser.

Honestly, I just want you to add these lists to the default filter so I don't have to add them manually:
https://easylist-downloads.adblockplus.org/fanboy-annoyance.txt
https://easylist-downloads.adblockplus.org/antiadblockfilters.txt

I've asked for it here #879 (comment)

[celenityy]

I once experienced a case where websites/domains could not be accessed after trying to add Hagezi Threat Intelligence Feeds and others (forgot which ones) to my OpenWRT router.

This is not intentional at all and I would highly recommend you report any false positives to Hagezi. I've personally never encountered any false positives from TIF, due to the careful curation & high quality sources. I'm not sure how long ago you used it, so it might be worth trying again, and again, please report any false positives if you experience them.

others (forgot which ones)

My guess would be the others are what caused your false positives, rather than TIF.

You need an admin if you want to use aggressive blocklists and that's not good for end users.
You just need “set it and forget it filters” for end user.

I generally agree, I think we should focus on enabling lists known to not cause breakage by default, and maybe we could also include some more aggressive lists, but just not enable them by default & leave to the end user to decide (Like ex. uBlock Origin & AdGuard do for instance). I feel like most of the lists I mentioned are generally safe and shouldn't cause issues, with the main exception of 1Hosts (Pro).

Also, those lists can simply be handled/managed by app such as rethinkdns, personaldnsfilter, invizible pro and others for system-wide not just the browser.

I agree that some of these lists are best used at a DNS level, but we can't count on the majority of average users having DNS protection in place (not to mention also having the specific desired list). Therefore, I think it'd be a good idea to still include some domain lists like this (Other content blockers such as uBlock Origin & AdGuard also include some domain lists), but I agree, we also need more advanced lists as well, such as potentially the uBlock Origin filters if they are compatible.

Honestly, I just want you to add these lists to the default filter so I don't have to add them manually:
https://easylist-downloads.adblockplus.org/fanboy-annoyance.txt
https://easylist-downloads.adblockplus.org/antiadblockfilters.txt

I agree with Anti-Adblock filters, but I'm a little mixed on Fanboy's Annoyances. I think the problem is "annoyances" are subjective by nature, ex. what one may deem as annoying, another may not. I think it'd be worth taking a closer look at what exactly they define as "annoying" to block, and then deciding from there. I personally always use Fanboy's Annoyances myself and haven't had issues, but again, I think it can be subjective depending on use case & personal preference.

[wildbeetle]

I agree with Anti-Adblock filters, but I'm a little mixed on Fanboy's Annoyances. I think the problem is "annoyances" are subjective by nature, ex. what one may deem as annoying, another may not. I think it'd be worth taking a closer look at what exactly they define as "annoying" to block, and then deciding from there. I personally always use Fanboy's Annoyances myself and haven't had issues, but again, I think it can be subjective depending on use case & personal preference.

As you said add to default doesn't mean enable by default.
Moreover, they are supplementary filter lists from easylist that are missing in the default list.

[celenityy]

As you said add to default doesn't mean enable by default.
Moreover, they are supplementary filter lists from easylist that are missing in the default list.

Ah, apologies @wildbeetle I misunderstood your point. I 100% agree that Fanboy's Annoyances should be added by default.

[uazo]

Oh no, please don't add those blocklists or enable them by default.

actually what I want is just to enable them by default, so that cromite appears to behave the same.
But before we do that, let's talk about it.

[wildbeetle]

Hello again @uazo @Retold3202
just an update, actually I used hagezi pro and hagezi TIF (full) on my router for several weeks without any problems.

Forgot to mention TIF IP too.

[celenityy]

Great to hear! :)

[gauze756]

I'd like to cast light on the domain count of these blocklists since Cromite is supported Android 7+
What i mean by that is this: With so many domains is the browser's adblock going to be a little bit too resource intensive for older devices?

Here are some of the Blocklist lists with their respective domain counts:

EasyList 85k+
EasyPrivacy 52k+
EasyList Cookie 17k+
Fanboy's Annoyances 84k+
HaGeZi's Threat Intelligence Feeds 643k+
HaGeZi's Multi Pro++ 187k+
Divested Combined Blocklist w/ Wildcards 659k+ w/o Wildcards 1.557.638
Dandelion Sprout's Anti-Malware List 102k+
Dandelion Actually Legitimate URL Shortener Tool 2.607
Badblock+ 6.999

Here are some resources to consider:

List of Blocklists Intended for Mobile Devices
HaGeZi's Multi Pro mini

We absolutely have to consider every user since Cromite is a web browser primarily used on wide variety of mobile devices.

[celenityy]

We absolutely have to consider every user since Cromite is a web browser primarily used on wide variety of mobile devices.

You're correct - I don't think that we should necessarily enable all of the lists I mentioned by default, but rather we can just add them as options for the user to decide. I definitely should've laid that out better though.

As far as for lists we should enable by default for every user - I think as a starting point, this should look something like:

• EasyList
• EasyPrivacy
• EasyList Cookie
• HaGeZi's Threat Intelligence Feeds (Especially important since we don't have Safe Browsing support)
• Actually Legitimate URL Shortener Tool

These are probably the most important lists IMO.

I'd be interested to see data benchmarking these though and how they can impact lower end devices, probably needs testing. My personal experience using these (+ more lists...) enabled in uBlock Origin on a Pixel 7 Pro was very solid without issues, but obviously that's a higher end device and we can't expect that to be everyone's experience.

[gauze756]

I think we can safely enable by default HaGeZi's Pro mini, Actually Legitimate URL Shortener Tool and maybe HaGeZi's TIF Medium??? I would like to add Badblock+ by default as well if it breaks no functionality. I would like you to better explain how Badblock+ would benefit users who already use better known lists.

I think Mull Browser's user base and especially it's maintainer could help us enormously on performance impact of lists. They have been using uBlock w/ default uBlock lists for years. @SkewedZeppelin

[celenityy]

I would like you to better explain how Badblock+ would benefit users who already use better known lists.

Sure. A lot of BadBlock’s benefit/appeal comes down to the fact that our lists are more comprehensive than a lot of others out there that try to avoid breakage at all costs - BadBlock tries to avoid breakage where possible, but if the ends justify the means and there’a a large privacy/security impact, then we will block what is needed.

Also worth noting that every entry is manually researched and reviewed by me first, unlike most lists that just compile together a bunch of other blocklists. It’s not the biggest list in the world due to this, but I guess it’s sort of a quality vs. quantity sort of thing. None of the BadBlock lists should be relied on as the sole blocklist, but I think they are nice as complementary lists to other more widely used ones.

I’d encourage you check out the README if you haven’t already, since it goes much more in depth than I am here - I’m probably not doing this justice. Please let me know if you think I should update it or change anything! Also happy to answer any other questions you have :)

I think Mull Browser's user base and especially it's maintainer could help us enormously on performance impact of lists. They have been using uBlock w/ default uBlock lists for years.

Good thinking. I also wonder how efficient Cromite’s ABP is compared to uBlock Origin on mobile…

[wildbeetle]

I'd be interested to see data benchmarking these though and how they can impact lower end devices, probably needs testing. My personal experience using these (+ more lists...) enabled in uBlock Origin on a Pixel 7 Pro was very solid without issues, but obviously that's a higher end device and we can't expect that to be everyone's experience.

The low end device I have now is only Redmi 9 with 3GB RAM.
No problem at all.

And the filters I use on cromite.

Default:

• EasyList.
• I don't care about cookies.
• EasyPrivacy.
• Anti Circumvention and Snippets.

Custom:

• AdBlock Warning Removal List.
• Fanboy's Annoyances.
• Hagezi's Multi Pro Mini.
• Hagezi's TIF Medium.

Of course there is no problem on my more powerful device (S23+). :)

[uazo]

I have been trying for some time BadBlock plus, excellent but lately unfortunately breaks a few sites (like creepjs which I use for testing).
When it happened I immediately thought it was my bug, but instead it was caused by the filter.
I think the best would be to have a way to tell if the breakage is caused by the filter rules rather than by bugs or blockage of cromite functionality.
to be added to #323

[celenityy]

unfortunately breaks a few sites (like creepjs which I use for testing).

My thinking behind blocking it was to prevent ex. 3rd party sites from abusing it for fingerprinting - I should probably adjust it to 3p since you're right that it is useful for testing...

[wildbeetle]

In my opinion, using hagezi filters as an additional list is enough.
He has a good reputation, his filters are widely used/available in popular software such as AdGuardHome or RethinkDNS.
Also in ad blockers on OpenWrt routers as far as I remember they almost always provide hagezi filters there, some make them default.
He is also very active, I remember someone joking that he eats block lists for breakfast. LoL

[Metrokoto]

Another list to consider is @Kees1958's most used trackers list, I have been using it alongside Ad-Void, Badblock Plus, and my ABP format forks of @yokoffing's lists without issue for a while now.

I also deactivated EasyList and EasyPrivacy, and added @ryanbr's Fanboy Ultimate list for wider coverage with less lists.

I use a similar setup with uBlock Origin on the Windows version of Cromite, FWIW, but YMMV.

See:
https://github.com/Metrokoto/filterlists-abp
https://github.com/Kees1958/W3C_annual_most_used_survey_blocklist

[Metrokoto]

While I am here, I thought I would also recommend some default user scripts, I have been using these without issue also;

I Don't Care About Cookies for Bromite by @xarantolus (on Android):
https://github.com/xarantolus/bromite-userscripts/releases/latest/download/idcac.user.js?raw=true

AdGuard Extra by AdGuard (on Windows and Android):
https://userscripts.adtidy.org/release/adguard-extra/1.0/adguard-extra.user.js

Remove URL tracking by @KokichaKolevTM (on Android):
https://github.com/KokichaKolevTM/My-UserScripts/blob/main/Remove%20URL%20tracking.user.js?raw=true

I also use BPC. Won't name it or link it here, as GitHub has repeatedly allowed malicious DMCAs against it, but IYKYK.

[gauze756]

Let's try to keep it simple, minimal and most importantly always prefer established, actively maintained, lists that are well tested by many users. I don't think uazo would try to test all these lists. Frankly I think it would be a massive waste of time FOR cromite project.

[uazo]

I don't think uazo would try to test all these lists

no, you're right, I won't.
I will rather think of a way to evaluate a list, but I still have to think about it.

Frankly I think it would be a massive waste of time FOR cromite project.

It is very important to have common predefined lists that work, so that we all look the same since are detectable by sites.

[gauze756]

Honestly study Brave Browser's default filter lists(link below) and enable those and be done with it. No need to reinvent the wheel and waste any more time.

https://github.com/brave/adblock-resources/blob/master/filter_lists/list_catalog.json
Search "default" to find everything you need.

[sankhababu]

I would like you to better explain how Badblock+ would benefit users who already use better known lists.

Sure. A lot of BadBlock’s benefit/appeal comes down to the fact that our lists are more comprehensive than a lot of others out there that try to avoid breakage at all costs - BadBlock tries to avoid breakage where possible, but if the ends justify the means and there’a a large privacy/security impact, then we will block what is needed.

Also worth noting that every entry is manually researched and reviewed by me first, unlike most lists that just compile together a bunch of other blocklists. It’s not the biggest list in the world due to this, but I guess it’s sort of a quality vs. quantity sort of thing. None of the BadBlock lists should be relied on as the sole blocklist, but I think they are nice as complementary lists to other more widely used ones.

I’d encourage you check out the README if you haven’t already, since it goes much more in depth than I am here - I’m probably not doing this justice. Please let me know if you think I should update it or change anything! Also happy to answer any other questions you have :)

I think Mull Browser's user base and especially it's maintainer could help us enormously on performance impact of lists. They have been using uBlock w/ default uBlock lists for years.

Good thinking. I also wonder how efficient Cromite’s ABP is compared to uBlock Origin on mobile…

After failure to block adblock warning in Cromite with https://easylist-downloads.adblockplus.org/antiadblockfilters.txt I've tried https://raw.githubusercontent.com/bogachenko/fuckfuckadblock/master/fuckfuckadblock.txt?_=rawlist.

It worked, but given problems with some other news streaming site.

Then I applied Badblock and seems it working better.

Even I had to disable Cromite's inbuilt easy privacy list as it blocking picture loading in some site, but with Bad block thease not happening.

And last of all, https://d3ward.github.io/toolz/adblock shows 97% blocking of contents, which was 85% with all inbuilt filters active.

May Uazo and other knowledgeable persons test more on Badblock.

Greetings.