Include web-discovery-project in Brave

[kkuehlz]

--
The Web Discovery Project is intended to make Brave Search more relevant and useful for everyone.

It is a strictly opt-in feature.

Data collection in Web Discovery Project is done in a privacy preserving fashion. By default, it discards search queries that are too long or suspicious looking (e.g. those that include phone numbers). It also discards odd URLs (e.g. those containing hashes), URLs of pages with a no-index flag, and pages that are not public or require any sort of authentication.

The system is designed so that no data received can be linked back to individuals or their devices. For a URL to be sent it needs to be visited independently by a large number of people. All data received is unlinkable, making it impossible to build profiles or sessions of Web Discovery Project contributors.

The PR is including the bundles that can be found on the Web discovery project public repo

Further materials,

• Updates of privacy policy
• Description of the Web Discovery Project
• Tech documentation

[stephendonner]

Verified PASSED using

Brave	1.31.62 Chromium: 94.0.4606.54 (Official Build) beta (x86_64)
Revision	c8191a1d5cccbf64e8fe7269043f8ace8d74dd05-refs/branch-heads/4606@{#1130}
OS	macOS Version 11.6 (Build 20G165)

Went through and verified the STR/Cases outlined via brave/brave-core#9381 (comment) as per the following:

• ensured that Web Discovery Project is disabled by default via brave://settings/search
• ensured that clicking on Learn more opens https://brave.com/privacy/browser/#web-discovery-project without any issues
• ensured that WDP.isRunning returns false when Web Discovery Project is disabled
• ensured that WDP.isRunning returns true when Web Discovery Project is enabled

WDP.isRunning = false	WDP.isRunning = true

• ensured that WDP.isRunning stays true if enabled after restarting the browser several times
• ensured that WDP.isRunning stays false if disabled after restarting the browser several times
• ensured that an error was displayed via the browser console when using WDP.modules['web-discovery-project'].background.webDiscoveryProject.patternsLoader.resourceWatcher.forceUpdate() but WDP.isRunning is false
• ensured that WDP.modules['web-discovery-project'].background.webDiscoveryProject.patternsLoader.resourceWatcher.forceUpdate() displayed the correct data when WDP.isRunning is enabled/appearing as true

Disabled Output	Enabled Output

• waited ~10min once WDP was disabled after confirming that WDP.isRunning is false and ensured that no calls to wdp.brave.com occurred

---

Verification passed on

Brave | 1.31.66 Chromium: 94.0.4606.61 (Official Build) beta (64-bit)
-- | --
Revision | 418b78f5838ed0b1c69bb4e51ea0252171854915-refs/branch-heads/4606@{#1204}
OS | Windows 10 Version 21H1 (Build 19043.1237)

Went through and verified the STR/Cases outlined via brave/brave-core#9381 (comment) as per the following:

• ensured that Web Discovery Project is disabled by default via brave://settings/search
• ensured that clicking on Learn more opens https://brave.com/privacy/browser/#web-discovery-project without any issues
• ensured that WDP.isRunning returns false when Web Discovery Project is disabled
• ensured that WDP.isRunning returns true when Web Discovery Project is enabled

WDP.isRunning = false	WDP.isRunning = true

• ensured that WDP.isRunning stays true if enabled after restarting the browser several times
• ensured that WDP.isRunning stays false if disabled after restarting the browser several times
• ensured that an error was displayed via the browser console when using WDP.modules['web-discovery-project'].background.webDiscoveryProject.patternsLoader.resourceWatcher.forceUpdate() but WDP.isRunning is false
• ensured that WDP.modules['web-discovery-project'].background.webDiscoveryProject.patternsLoader.resourceWatcher.forceUpdate() displayed the correct data when WDP.isRunning is enabled/appearing as true

Disabled Output	Enabled Output

• Open a new tab in the browser and visit the following Google SERP: https://www.google.com/search?q=joker
• Close the tab once the page has fully loaded and go back to the console devtool.
• Run the following command:
  WDP.modules['web-discovery-project'].background.webDiscoveryProject.strictQueries.map(x=>x.tDiff=0)
• Wait up to ~20 seconds and you should see network requests being emitted like in the screenshot below (one or more calls to https://collector.wdp.brave.com/.)
  
• waited ~10min once WDP was disabled after confirming that WDP.isRunning is false and ensured that no calls to wdp.brave.com occurred

---

Verified PASSED using

Brave	1.31.66 Chromium: 94.0.4606.61 (Official Build) beta (64-bit)
Revision	418b78f5838ed0b1c69bb4e51ea0252171854915-refs/branch-heads/4606@{#1204}
OS	Linux

Went through and verified the STR/Cases outlined via brave/brave-core#9381 (comment) as per the following:

• ensured that Web Discovery Project is disabled by default via brave://settings/search
• ensured that clicking on Learn more opens https://brave.com/privacy/browser/#web-discovery-project without any issues
• ensured that WDP.isRunning returns false when Web Discovery Project is disabled
• ensured that WDP.isRunning returns true when Web Discovery Project is enabled

WDP.isRunning = false	WDP.isRunning = true

• ensured that WDP.isRunning stays true if enabled after restarting the browser several times
• ensured that WDP.isRunning stays false if disabled after restarting the browser several times
• ensured that an error was displayed via the browser console when using WDP.modules['web-discovery-project'].background.webDiscoveryProject.patternsLoader.resourceWatcher.forceUpdate() but WDP.isRunning is false
• ensured that WDP.modules['web-discovery-project'].background.webDiscoveryProject.patternsLoader.resourceWatcher.forceUpdate() displayed the correct data when WDP.isRunning is enabled/appearing as true

Disabled Output	Enabled Output

• waited ~10min once WDP was disabled after confirming that WDP.isRunning is false and ensured that no calls to wdp.brave.com occurred