What to do about Manifest V3

[olizilla]

As flagged by @dignifiedquire the proposed webextention api changes under the banner of the v3 manifest work will reduce the scope of what ipfs-companion is able to do to bring ipfs to the browser

Proposed changes: https://docs.google.com/document/d/1nPu6Wy4LWR66EFLeYInl3NzzhHzc-qnk4w4PX-0XMw8/edit?usp=sharing

Tracking issue: https://bugs.chromium.org/p/chromium/issues/detail?id=896897&desc=2

Most significant is the move away from the webRequest api to the declarativeNetRequest api
https://developer.chrome.com/extensions/declarativeNetRequest

The webRequest api is blocking. Chromium asks the webextention what to do about each outbound http request at runtime. IPFS Companion makes use of this to let the user toggle between redirecting ipfs addresses to a local daemon or the public gateway.

The declarativeNetRequest requires extentions to tell chromium in advance, what urls patterns they wish to block or redirect by defining them statically as rules in the manifest.json

There may well be other important changed for us in that doc, but this one stands out as the most significant.

[lidel]

Oh wow, a lot to unpack here.

After reading the draft with proposed changes I wrote down some highlights with interesting changes and potential pain points (below).

Will follow V3 as it unfolds, but it is very likely it will bring us additional maintenance cost.

Restrict the blocking capabilities of the webRequest API

In Manifest V3, this API will be discouraged (and likely limited) in its blocking form. The non-blocking implementation of the webRequest API, which allows extensions to observe network requests, but not modify, redirect, or block them (and thus doesn't prevent Chrome from continuing to process the request) will not be discouraged. As an alternative, we plan to provide a declarativeNetRequest API . The details of what limitations we may put in the webRequest API are to be determined. [..]

• While deciding whether a request is to be blocked or redirected, the declarativeNetRequest API is given priority over the webRequest API because it allows for synchronous interception.
• Evaluation of declarativeNetRequest rulesets is performed during the onBeforeRequest stage of a network request.

We use blocking capability of webRequest API to inspect headers, detect IPFS resources, do DNSLink lookups and other things – src/lib/ipfs-companion.js#L95-L97:

browser.webRequest.onBeforeSendHeaders.addListener(onBeforeSendHeaders, { urls: ['<all_urls>'] }, ['blocking', 'requestHeaders'])
browser.webRequest.onBeforeRequest.addListener(onBeforeRequest, { urls: ['<all_urls>'] }, ['blocking'])
browser.webRequest.onHeadersReceived.addListener(onHeadersReceived, { urls: ['<all_urls>'] }, ['blocking', 'responseHeaders'])

Personally I don't think this change will stick in the form proposed in the draft as having declarativeNetRequest as the only blocking option makes it impossible to write things like uBlock Origin or uMatrix, which could create terrible PR for Google, given their Ad business.

If the worst comes to the worst, for the basic functionality we may be okay with declarativeNetRequest and some elaborate rules for redirecting https?://*/ip(f|n)s/* and https?://*.ip(f|n)s.*/, but supporting DNSLink in current seamless form may be difficult/impossible (or would require user action/opt-in per site).

Replace background pages with ServiceWorkers

We plan to replace background pages with ServiceWorkers, registered at install time and granted access to extension APIs. [..]
We will be able to reduce the amount of extension-specific code since, instead of implementing a custom background-running presence, we can extend the existing ServiceWorker context. Most importantly, the platform will evolve with the open web; as improvements to ServiceWorkers and the web platform are made, the extensions platform benefits from those same improvements.

We need a stable, persistent background presence to be able to run embedded js-ipfs node.
It does not sound like SeviceWorkers, which is often killed by the browser when idle.
I hope there will be an inexpensive and reliable way to keep SW alive in WebExtension context.

If not, @Gozala had some ideas on keeping ServiceWorker alive in ipfs/in-web-browsers#137 (comment), but the net result will be to actually use more CPU by artificially keeping it alive than when having a background page.

Restricting Origin Access

In Manifest V3, host permissions will be granted by the user at runtime (similar to activeTab, but with options for the user to choose to always run on a specific site or all sites), rather than install-time.

In Manifest V3, we want activeTab-style host permissions to be the default, with a number of extra options. Instead of being granted access to all URLs on installation, extensions will be unable to request <all_urls>, and instead the user can choose to invoke the extension on certain websites, like they would with activeTab. Additional settings will be available to the user post-installation, to allow them to tweak behavior if they so desire.

[..] In the default case (click-to-run), it is clear to the user when the extension is running, and has a safe default (not running on any site). When the user chooses to invoke the extension on a given site, there is implicit understanding that the extension will "see" the contents of the page.

This sounds bad, as it suggests by default it will be click-to-run, but in later section about host permissions support for <all_urls> is described as open question:

Open question: Should activeTab be removed in favor of specifying <all_urls> in the host_permissions key of the manifest?

Also, this blogpost from 2018 suggests how the UI will look like:

Beginning in Chrome 70, users will have the choice to restrict extension host access to a custom list of sites, or to configure extensions to require a click to gain access to the current page.

Cross-Origin Communication

Beginning in Manifest V3, content scripts will not be given special privileges regarding the requests they can make. If a content script needs access to data fetched from a cross-origin server that requires authentication, it can proxy the request through its background page using extension messaging.

As long they limit this restriction to Content Scripts, we should be fine.

We do CORS-but-not-really to IPFS API only from the background page. "Not really" because we remove Origin header from requests made to API URL by js-ipfs-http-client running in WebExtension context to remove need for manual whitelisting via Access-Control-Allow-Origin at go-ipfs: src/lib/ipfs-request.js#L119-L151.

Promise-Based APIs

Extension APIs will be promise-based. The older callback version will continue to be supported.

I was about to say its cool, as there will be no need for mozilla/webextension-polyfill, but we will probably need a polyfill for all the other breaking changes. Time will tell, so far issues to follow:

• Migrate apps/extension APIs to use Promises instead of callbacks
• Extension APIs should be also exposed under browser.* to match the WebExtensions spec

Dynamic Content Scripts (P2)

we can allow extensions to dynamically add and remove, or enable and disable, content scripts. This would allow extensions to only add these scripts once they have permission to do so.

There are additional situations in which this is beneficial, as well. Currently, the advice for extensions wishing to dynamically inject scripts based on some knowledge at runtime is to use the tabs.executeScript API; however, this is insufficient for certain use cases. In particular, this cannot (reliably) insert a script before the page finishes loading, which is a feature that content scripts provide. Allowing dynamic content scripts would solve this use case.

This is good news, as it would solve issue with window.ipfs described in #362 (comment)

(browser.contentScripts works in Firefox, but it's not implemented in Chrome yet)

Refactoring of various APIs

Additional work and duplicated code, but nothing critical.

Sad realization: even if Firefox follows these V3 API changes, for some time it won't, while Chromium will, so it will be like writing websites in 90s with similar level of code duplication for achieving the same thing in two different browsers :|

---

ps. what a perfect issue number 👌

[Gozala]

If not, @Gozala had some ideas on keeping ServiceWorker alive in ipfs/in-web-browsers#137 (comment), but the net result will be to actually use more CPU by artificially keeping it alive than when having a background page.

Note that keeping SW alive is going to be an arms race with browsers. Also was proposing for Safari (As replacement for SharedWorker due to lack of it) is different as you'll have a clients (documents served from it) which is considered as legitimate case.

[Gozala]

Personally I don't think this change will stick in the form proposed in the draft as having declarativeNetRequest as the only blocking option makes it impossible to write things like uBlock Origin or uMatrix, which could create terrible PR for Google, given their Ad business.

Not so sure about it, they might ship their own content blocker (that would likely don't harm their Ad business) and call it a day
https://bugs.chromium.org/p/chromium/issues/detail?id=896897&desc=2#c23

[Gozala]

I'm not sure if there are some way to engage with the corresponding people driving these changes but I think best bet would be to convince them to expose SharedWorker (or equivalent) for long lived tasks or maybe an official API that would provide a way to keep ServiceWorker alive.

[lidel]

Extensions Developer Advocate at Chrome is doing office hours!
I grabbed the very first free slot (April 12, 16:30 UTC).

My plan is to give an overview of what ipfs-companion does, potential API improvements and our concerns around Manifest V3.

Update: We gave a brief demo and an overview of our needs during the office hours.

There is still a lot of unknowns around Manifest V3 (especially no details on if/how Service-Worker-like API is going to replace JS process running in persistent background pages).

As for webRequest, activeTab and <all_urls> Privacy Badger team has written up some relevant requests for changes: Chrome team requests-Privacy Badger.pdf

We need to wait until another draft is published to see if our concerns are adressed.

[lidel]

Some updates/clarifications from Simeon (Extensions Developer Advocate):
https://groups.google.com/a/chromium.org/d/msg/chromium-extensions/veJy9uAwS00/9iKaX5giAQAJ

[.. ] I want to emphasize that the Manifest V3 design document is not exhaustive or immutable. The extensions team is pursuing the goals outlined in this design document and iterating on design and implementation details. The best way for developers to really understand the changes is to experiment with the Manifest V3 platform.

[..] the extensions team is currently working on a Developer Preview of Manifest V3. Our goal with this preview is to give developers a way to start experimenting with some of the most significant changes to the platform in order to provide targeted feedback.

When they ship Developer Preview, we will prototype a port and see how V3 impacts Companion in practice, but based on analisys from Raymond Hill (uBlockOrigin) I would not be too optimistic:

The blocking ability of the webRequest API is still deprecated, and Google Chrome's limited matching algorithm will be the only one possible, and with limits dictated by Google employees

It's annoying that they keep saying "the webRequest API is not deprecated" as if developers have been worried about this -- and as if they want to drown the real issue in a fabricated one nobody made.
– uBlockOrigin/uBlock-issues#338 (comment)

[lidel]

Response from the Ghostery team suggests the manifest v3 performance claim does not hold.
https://whotracks.me/blog/adblockers_performance_study.html
tl;dr degradation is milliseconds at worst

[lidel]

Update from Google https://blog.chromium.org/2019/06/web-request-and-declarative-net-request.html

Chrome provides enterprise controls through its administrator policies. The blocking version of the Web Request API remains available for managed extensions because of the deep integrations that enterprises may have between their software suites and Chrome.

Pretty bad news, means current onBeforeRequest it won't work out of the box and we need to re-implement entire redirect logic using "The Declarative Net Request API"

The Declarative Net Request API now allows for the registration and removal of dynamic rules - specified at runtime rather than statically in the manifest. We’ve also added the capability to remove common tracking headers, such as Referer, Cookie, and Set-Cookie.

In our case the devil is in the details.

Setting up a crude redirect of /ipfs/{cid} paths with this API should work fine. When changing gateway Companion would remove/add new redirect rule for /ipfs/* paths.
Problem: we lose ability to do CID validation, so we need to work round false-positives.

We could do DNSLink check in non-blocking onBeforeRequest and add redirect rules based on that (similar to current best-effort strategy, but it will continue to yield false-negatives on initial request.

We may lose ipfs:// handler done via search-query hack, so would be great if real handler landed before that happens:

• https://bugs.chromium.org/p/chromium/issues/detail?id=651311
• https://bugs.chromium.org/p/chromium/issues/detail?id=64100#c12

Shall see when Developer Preview lands.

---

Related discussion thread: https://groups.google.com/a/chromium.org/forum/#!msg/chromium-extensions/qFNF3KqNd2E/8R9PWdCbBgAJ

[lidel]

A number of extension developers have reached out to ask how Mozilla plans to respond to the changes proposed in v3. Following are answers to some of the frequently asked questions.

Mozilla’s Manifest v3 FAQ

[lidel]

ipfs-companion/issues/808: Permanent in-depth review on Chrome Web Store:

Google makes it more and more difficult to publish powerful extensions such as IPFS Companion.

In preparation for Manifest v3 (#666) Chrome Web Store artificially slows down publishing of extensions that requesti access to certain APIs.

[lidel]

Chrome team rolled out Manifest V3 into Canary:
https://groups.google.com/a/chromium.org/forum/#!msg/chromium-extensions/hG6ymUx7NoQ/TiAe0gI5AQAJ

In order to help developers start experimenting with and transitioning to MV3 we've put together a migration guide. This doc provides a high-level look at what's changing and what developers will need to do to adapt. We also have a guide specifically for migrating from background pages to service workers.

I will dig into this deeper next week, but from a quick read it seems we need to refactor most of Companion to support both V2 (Brave, Firefox) and V3 (Google Chrome, future Chromium). It is unclear if we will be able to persist an embedded js-ipfs node in V3 at all.

Quick API Checklist vs Impact on IPFS Companion (:ballot_box_with_check: means YES)

• ☑️ Do you have host permissions in your permissions or optional_permissions arrays in manifest.json?
• ☑️ Are you using background pages?
• ☑️ Are you using the browser_action or page_action property in manifest.json?
• ☑️ Are you using the chrome.browserAction or chrome.pageAction JavaScript API?
• ☑️ Are you currently using the blocking version of chrome.webRequest in a consumer-facing extension?
• ☑️ Are you currently using chrome.tabs.executeScript({code: '...'}), eval(), or new Function() in background contexts or content scripts?
• ☑️ Are you currently using chrome.runtime.getBackgroundPage(),
• 🤷‍♂️ Are you making CORS requests in content scripts?
• ☑️ Are you using a custom content_security_policy in manifest.json?

[lidel]

Redirects with V3 are broken at the moment:

• Issue 1024211: Manifest V3: webRequest listeners not called after service worker stops

[lidel]

• Google published migration guide
• Microsoft Edge is preparing for Manifest V3 as well.

Manifest V3 now seems inevitable. After we wrap ongoing work in Q4, should build a PoC.

[lidel]

Adblocker Dev Summit 2020

• Some history of ad blocking and limitations of declarative filtering approach on Apple platforms: https://www.youtube.com/watch?v=_dduavvDj6s
• Status of Manifest v3 in Firefox: https://www.youtube.com/watch?v=tpDFS-GUytg
• Status update on Chrome side: https://www.youtube.com/watch?v=zxWG2yUcTFA

Originally posted in uBlockOrigin/uBlock-issues#338 (comment)

[lidel]

Google, Microsoft, Apple, and Mozilla have launched the WebExtensions Community Group (WECG) to collaborate on standardizing browser extensions:

• https://www.bleepingcomputer.com/news/security/google-microsoft-and-mozilla-work-together-on-better-browser-extensions/
• https://github.com/w3c/webextensions/blob/main/charter.md

Working towards removing annoying differences between vendors is a good news, it also means the direction of "Manifest V3" is no longer controlled by Google on their own, and Microsoft's experiments with PWA and protocol handlers have some overlap with our use cases.

Thin on details, time will tell if we end up in better place.

[lidel]

It has begun: https://developer.chrome.com/docs/extensions/mv3/intro/mv3-migration/

[lidel]

HN discussion: AdGuard publishes the first ad blocker built on Manifest V3

[lidel]

Timeline changed! Google postpones MV2 shutoff in Chrome stable to June 2023:

• Starting in January in Chrome 112, Chrome may run experiments to turn off support for Manifest V2 extensions in Canary, Dev, and Beta channels.
• Starting in June in Chrome 115, Chrome may run experiments to turn off support for Manifest V2 extensions in all channels, including stable channel.
• In January 2023, use of Manifest V3 will become a prerequisite for the Featured badge as we raise the security bar for extensions we highlight in the store.
• In June 2023, the Chrome Web Store will no longer allow Manifest V2 items to be published with visibility set to Public. All existing Manifest V2 items with visibility set to
  Public at that time will have their visibility changed to Unlisted.

– https://developer.chrome.com/blog/more-mv2-transition/

HN discussion: https://news.ycombinator.com/item?id=33012057

[lidel]

Mozilla finally started accepting MV3 on their store starting November 21:

Starting November 21, 2022 add-on developers are welcome to upload their Firefox Manifest version 3 (MV3) compatible extensions to addons.mozilla.org (AMO) and have them signed as MV3 extensions.

• https://blog.mozilla.org/addons/2022/10/31/begin-your-mv3-migration-by-implementing-new-features-today/
• https://blog.mozilla.org/addons/2022/11/17/unified-extensions-button-and-how-to-handle-permissions-in-manifest-v3/
• https://blog.mozilla.org/addons/2022/11/17/manifest-v3-signing-available-november-21-on-firefox-nightly/

This is good news, means for the first time we can make a MV3 release and publish the same code to both Mozilla and Google.

[whizzzkid]

@tinytb @lidel https://groups.google.com/a/chromium.org/g/chromium-extensions/c/zQ77HkGmK9E

We get some breathing room here, pushed to March '23

Thanks @ricmk

[SgtPooki]

The work we're doing for mv3 has been mostly encapsulated into a singular issue at #1152

@whizzzkid do we still need this issue?

[whizzzkid]

Update on March 29, 2023: https://groups.google.com/u/0/a/chromium.org/g/chromium-extensions/c/zQ77HkGmK9E

Looks like MV3 deadline has been moved forward.

[BigLep]

@whizzzkid : what are the conditions for closing this issue? We have an answer for "what to do about Manifest v3" that is being tracked in #1152 . Can we resolve this and have the updates related to the rollout occur there?

[whizzzkid]

Implementation Plan exists: #1152

Work is in progress!

[lidel]

Work happening in #1182, let's keep this open until it lands in main.

[lidel]

For anyone still watching this issue for updates on Manifest V3,
Google resumed the Manifest V2 deprecation plan.

We will begin disabling Manifest V2 extensions in pre-stable versions of Chrome (Dev, Canary, and Beta) as early as June 2024, in Chrome 127 and later.

Enterprises using the ExtensionManifestV2Availability policy to ensure the continued functioning of Manifest V2 extensions in their organization will have one additional year - until June 2025 - to migrate the Manifest V2 extensions in their organization.

More details in https://developer.chrome.com/blog/resuming-the-transition-to-mv3/