Add AdGuard's $network support on Firefox

[Yuki2718]

Prerequisites

• I verified that this is not a filter list issue. Report any issues with filter lists or broken website functionality in the uAssets issue tracker.
• This is not a support issue or a question. For support, questions, or help, visit /r/uBlockOrigin.
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue.
• The issue is not present after disabling uBO in the browser.
• I checked the documentation to understand that the issue I am reporting is not normal behavior.

I tried to reproduce the issue when...

• uBO is the only extension.
• uBO uses default lists and settings.
• using a new, unmodified browser profile.

Description

Documentation: https://adguard.com/kb/general/ad-filtering/create-own-filters/#network-modifier
Related issue: AdguardTeam/tsurlfilter#104

A specific URL where the issue occurs.

https://github.com/AdguardTeam/AdguardFilters/pull/160415/files

Steps to Reproduce

Check the links above

Expected behavior

NA

Actual behavior

NA

uBO version

1.51.0

Browser name and version

Firefox 116.0.3

Operating System and version

Windows 10

[gwarser]

Related #1070 (comment)

Well it's possible to block at onHeadersReceived() time and the ip address information is available at this point

Will blocking on response suffice?

[Yuki2718]

If can then display strict-blocking page.

[DandelionSprout]

Goodness knows how this would be implemented in practice, but $network is fantastic stuff when done right, especially to handle IPs known to host sometimes thousands of malware domains and only 0~2 legitimate domains.

[JobcenterTycoon]

This would also be useful to block rotating propeller adservers in real time.

We should also be able to add additional modifiers like ||127.0.0.1^$network,script,3p this would allow users to fine control their traffic which can't be setup with a regular firewall.

[gorhill]

Related commit (forgot to link issue in commit message): gorhill/uBlock@c6dedd2

[JobcenterTycoon]

Ok i tested on https://woxikon.in/ and the filter *$script,xhr,ipaddress=139.45.197.170 successfully blocked the Propeller request https://sswpawbv.xyz/

[D4niloMR]

It's possible to ipaddress block document? header too, would be good if possible.

[gorhill]

Real use case?

I did look at whether document should be blocked but it turned out to be a bit more work then I expected so I left this out.

[D4niloMR]

I was thinking about those domains from badware that have a lot of similar domains in the same ip https://viewdns.info/reverseip/?host=m8u9f3.com&t=1, https://viewdns.info/reverseip/?host=172.64.155.33&t=1

Though we would need to be very careful to not block legitimate sites.

[DandelionSprout]

I did some testing for 40min tonight in Firefox 130.0 with uBO 1.51.1b17. I don't intend to put immediate pressure on any involved parties, I'm simply listing them.

• I was (with some difficulty) able to get $ipaddress= to work with IPv6 from as a proof-of-concept. The supported syntax is without square brackets, resulting in *$all,ipaddress=2a00:1450:400f:803::200e. Getting it to work on ipv6.google.com required more than one ping ipv6.google.com in PowerShell to see which IPs the many involved domains had, and getting it to work on IPv6+IPv4 sites (I tested with www.dr.dk) would be more than a little difficult.
• Although $all is supported as a modifier, which results in e.g. *$all,ipaddress=199.232.42.217, the modifier does not seem to have any additional effects compared to having no additional modifiers.
• Requests that can be blocked with $ipaddress include script, xhr, image, frame, css, and beacon (I was unable to test with media in this hurry).

• "I did look at whether [$doc] should be blocked but it turned out to be a bit more work then I expected so I left this out."
Fair enough.

[gorhill]

Somehow I thought Firefox's ip field was using brackets for ipv6, turns out it's not. Will fix.

---

Fixed in 1.59.1b18

[JobcenterTycoon]

When i open https://720pflix.fit/ it shows the first propeller request as unblocked but i see no requests in the Firefox network logger. After a reload its fine.

[gorhill]

It's normal, the first time the IP address is not available in uBO's DNS cache, so the first call to onBeforeRequest() will log a non-matched entry. Once the the DNS resolution occurs, there will be another call to onBeforeRequest but with the IP address available, which will then match your filter. It works like this:

• OnBeforeRequest
  • IP address available?
    • Yes:
      • Add ip address to request details
      • Perform request matching (log entry)
    • No:
      • Fetch it through an asynchronous DNS query
        • DNS query fulfilled
        • Add ip address to request details
        • Perform request matching (log entry)
      • Perform request matching (log entry)

It's also how matching against cname has been working, there is always two entries in the logger when there is a cname. The logger really reports request matching calls, and there can be more than one. I suppose it would be useful to show at which stage the matching is made so that we know whether there was an actual request reaching the remote server, but there is not much horizontal space for more details in the logger.

[garry-ut99]

Having 2 filters with several hundreds of excluded domains (simplified example):

*$doc,domain=~domain1.com|~domain2.org|~domain3.net

• where the left part is redacted and replaced with * for the purpose of example
• and the right part was a list of hundreds of excluded domains
• (and where I provide only two example IPs in below examples as well)

and trying to reduce lenght and complexity of the filters by replacing several hundreds of excluded domains by a small bunch of excluded ipaddress IPs which cover majority of domains (several hundreds of domains are sticked to only several IPs), but it turned out that:

Neither negating nor pipeing works (not sure whether it's unsupported or a bug):

*$doc,ipaddress=~95.216.7.22
*$doc,ipaddress=95.216.7.22|9.9.9.9
*$doc,ipaddress=95.216.7.22|~9.9.9.9
*$doc,ipaddress=~95.216.7.22|~9.9.9.9

Pipeing does work in regex (but sometimes might be a bit cumbersome) but negating works only partially, can't find 100% working regex solution:

*$doc,ipaddress=/^(?!95\.216\.7\.22|9\.9\.9\.9).*$/
*$doc,ipaddress=/^((?!95\.216\.7\.22|9\.9\.9\.9).)*$/

• the first filter doesn't match 95.216.7.22y where y is any digit, for example 95.216.7.228
• the second filter even worse: doesn't match x95.216.7.22y where x and/or y is any digit, for example 195.216.7.22 or 95.216.7.221 or 195.216.7.221.

(Firefox)

[gorhill]

~ and | is not supported. The value is not validated, so you can enter anything. Only one ipv4 or ipv6 address or regex is valid.

[garry-ut99]

Good to know, as this information wasn't mentioned anywhere, by the way, would be good to know also:

• whether the answer means also that the functionality won't be implemented
• whether there exists a regex solution which can accomplish the task at 100%

(I also assume that possibly the former might depend on the latter)

[gorhill]

Negation and multi-address won't be implemented, there is no actual real world use cases for this.

accomplish the task at 100%

What task?

[garry-ut99]

Not that I'm saying it must be implemented, but just simply replying: they were real use cases, which I have in "My filters", just that more for personal usage than to land in public filter lists, that's why I redacted and simplified them for the sake of example. The left part was a long regex (highlighted by yellow by uBO linter) with many unwanted phrases that might exist in a domain name, and the right part was a list of several hundred of excluded domains to not trigger the long regex on most common sites + other sites I visit. The purpose of the filters is to not visit some subcategories of categories of sites. But unless there is more of real usage cases (especially ones that could land in public lists) then yes, we can live without it.