Include web discovery bundles in the brave extension

[kkuehlz]

We'll vendor the code at sync time until we can port their build system

Resolves brave/brave-browser#18166

Test Plan

See following comment: #9381 (comment)

Submitter Checklist:

• I confirm that no security/privacy review is needed, or that I have requested one
• There is a ticket for my issue
• Used Github auto-closing keywords in the PR description above
• Wrote a good PR/commit description
• Added appropriate labels (QA/Yes or QA/No; release-notes/include or release-notes/exclude; OS/...) to the associated issue
• Checked the PR locally: npm run test -- brave_browser_tests, npm run test -- brave_unit_tests, npm run lint, npm run gn_check, npm run tslint
• Ran git rebase master (if needed)

Reviewer Checklist:

• A security review is not needed, or a link to one is included in the PR description
• New files have MPL-2.0 license header
• Adequate test coverage exists to prevent regressions
• Major classes, functions and non-trivial code blocks are well-commented
• Changes in component dependencies are properly reflected in gn
• Code follows the style guide
• Test plan is specified in PR before merging

After-merge Checklist:

• The associated issue milestone is set to the smallest version that the
  changes has landed on
• All relevant documentation has been updated, for instance:
  • https://github.com/brave/brave-browser/wiki/Deviations-from-Chromium-(features-we-disable-or-remove)
  • https://github.com/brave/brave-browser/wiki/Proxy-redirected-URLs
  • https://github.com/brave/brave-browser/wiki/Fingerprinting-Protections
  • https://github.com/brave/brave-browser/wiki/Brave%E2%80%99s-Use-of-Referral-Codes
  • https://github.com/brave/brave-browser/wiki/Custom-Headers
  • https://github.com/brave/brave-browser/wiki/Web-Compatibility-Exceptions-in-Brave
  • https://github.com/brave/brave-browser/wiki/QA-Guide
  • https://github.com/brave/brave-browser/wiki/P3A

Test Plan:

[remusao]

Some of the bundles seem to be missing (they are not injected directly from manifest.json but are needed to initialize instances of Worker dynamically):

• rusha.bundle.js which can be found in the output folder at build/modules/human-web/rusha.bundle.js. It is needed to initialize a Worker dynamically here: https://github.com/brave/humanweb/blob/main/modules/human-web/sources/human-web.es#L5399
• worker.wasm.bundle.js and worker.asmjs.bundle.js which can both be found in build/modules/hpnv2/ and are needed to initialize a Worker here: https://github.com/brave/humanweb/blob/main/modules/hpnv2/sources/group-signer.es#L56

I think we might need to figure out the value of this config.baseURL which is needed to find the path to the bundle at runtime (and enable the workers).

Optionally, these bundles come with sourcemaps, but maybe we can ignore that until we do some more build system work and move to the webpack stack.

[petemill]

Should we at least put a more unobfuscated build / bundle output from the current humanweb source here?
Also if we really want the source in this repo (or as a DEP) in the short-term before we convert the humanweb code to webpack it would actually be pretty easy to make and have a build step which calls npm run build-humanweb and just run the same command humanweb does in its current repo.

[remusao]

Should we at least put a more unobfuscated build / bundle output from the current humanweb source here?
Also if we really want the source in this repo (or as a DEP) in the short-term before we convert the humanweb code to webpack it would actually be pretty easy to make and have a build step which calls npm run build-humanweb and just run the same command humanweb does in its current repo.

Given the amount of work for v0 and unless there is strong opposition to this idea, I'd ideally prefer to keep the code in a separate repo for simplicity. I think there are also few advantages in having it separated for now (and open-source, of course):

1. Single purpose repository which is independent from browser (it's "just" an extension and the scope is easier to reason about).
2. CI and build/test flows already work (it would need to be ported/adapted to brave-core).
3. Easier for people looking for technical documentation about human-web to find it.

I'm sure all of these can be solved but there are many other blocking items on the list of things to do.

Wdyt?

[diracdeltas]

Given the amount of work for v0 and unless there is strong opposition to this idea, I'd ideally prefer to keep the code in a separate repo for simplicity. I think there are also few advantages in having it separated for now (and open-source, of course):

That is fine as long as people can verify that the bundled code is the same as the code in the repo, like via a reproducible build process that is documented in the open source repo.

[diracdeltas]

Instead of including the bundled code in the git tree, you could have a build time script that pulls the open source code from the separate repo and bundles it, for instance.

[remusao]

Instead of including the bundled code in the git tree, you could have a build time script that pulls the open source code from the separate repo and bundles it, for instance.

I guess we can publish releases from the open-source repository and include them in package.json/package-lock.json right? Then people know which tag is currently in use in Brave.

[kkuehlz]

Instead of including the bundled code in the git tree, you could have a build time script that pulls the open source code from the separate repo and bundles it, for instance.

@diracdeltas did this in the newest push

[kkuehlz]

@remusao I added the copywriter's text and the entry to the settings page, all tied up to extension enable/disable.

[remusao]

Nit. Since you already check if (pref ...) in the toggleWebDiscovery function maybe this can be simplified with:

Suggested change

	const pref = prefs.find(p => p.key === WEB_DISCOVERY_PREF_KEY)
	if (pref) {
	toggleWebDiscovery(pref)
	}
	toggleWebDiscovery(prefs.find(p => p.key === WEB_DISCOVERY_PREF_KEY))

[atuchin-m]

I've pushed a patch from @remusao in the slack chat.

[mkarolin]

chromium_src ++

[bridiver]

This is actually very late in the process. As discussed on slack I think we want webRequest.onCommitted instead

[remusao]

Testing guidelines for QA:

1. Launch browser and go to Settings.
2. Check 1: Filter prefs by Web Discovery and check that there is a Web Discovery Project preference available:

3. Check 2: The preference should be disabled by default.
4. Open a new tab and visit brave://inspect/#extensions.
5. Click on inspect below the line with name Brave

6. Then visit the console tab (there are a bunch of logs which you can ignore as they come from other features of the Brave extension).
7. Check 3: Enter the following expression in the console prompt: WDP.isRunning and press enter. You should see false being displayed.

8. Now go back to the "Settings" page and turn Web Discovery Project on.
9. Check 4: Entering WDP.isRunning again in the console should now display true.
10. Check 5: Restarting the browser, should preserve the setting (it should stay true after restart; same after opting-out => it should stay false after restart once we disable Web Discovery Project).
11. Back to the console window, select the cog on the right (bottom one) then select "Log XMLHttpRequests" (this will allow us to see network requests from the console directly.

12. Enter the following command in the prompt:

WDP.modules['web-discovery-project'].background.webDiscoveryProject.patternsLoader.resourceWatcher.forceUpdate()

13. You should see some network requests like this:
    

14. Then open a new tab in the browser and visit the following Google SERP: https://www.google.com/search?q=joker

15. Close the tab once the page has fully loaded and go back to the console devtool.

16. Run the following command:

WDP.modules['web-discovery-project'].background.webDiscoveryProject.strictQueries.map(x=>x.tDiff=0)

17. Check 6: Wait up to ~20 seconds and you should see network requests being emitted like in the screenshot below.

The important bit here is the first call to similar URL you visited in the tab (double-fetch), then one or more calls to https://collector.wdp.brave.com/.

18. In Settings, opting-out then restarting the browser, then running WDP.isRunning in console should show false. There should be no network requests seen for wdp.brave.com in the console.

[kjozwiak]

Verification PASSED on Win 10 x64 using the following build:

Brave | 1.32.20 Chromium: 94.0.4606.54 (Official Build) nightly (64-bit)
-- | --
Revision | c8191a1d5cccbf64e8fe7269043f8ace8d74dd05-refs/branch-heads/4606@{#1130}
OS | Windows 10 Version 20H2 (Build 19042.1237)

Went through and verified the STR/Cases outlined via #9381 (comment) as per the following:

• ensured that Web Discovery Project is disabled by default via brave://settings/search
• ensured that clicking on Learn more opens https://brave.com/privacy/browser/#web-discovery-project without any issues
• ensured that WDP.isRunning returns false when Web Discovery Project is disabled
• ensured that WDP.isRunning returns true when Web Discovery Project is enabled

WDP.isRunning = false	WDP.isRunning = true

• ensured that WDP.isRunning stays true if enabled after restarting the browser several times
• ensured that WDP.isRunning stays false if disabled after restarting the browser several times
• ensured that an error was displayed via the browser console when using WDP.modules['web-discovery-project'].background.webDiscoveryProject.patternsLoader.resourceWatcher.forceUpdate() but WDP.isRunning is false
• ensured that WDP.modules['web-discovery-project'].background.webDiscoveryProject.patternsLoader.resourceWatcher.forceUpdate() displayed the correct data when WDP.isRunning is enabled/appearing as true

Disabled Output	Enabled Output

• waited ~10min once WDP was disabled after confirming that WDP.isRunning is false and ensured that no calls to wdp.brave.com occurred