Google components automatic update fails #10464

jumde · 2020-06-25T02:59:10Z

Description

Automatic updates for google components fail.

Steps to Reproduce

Navigate to brave://components
Wait for a few minutes

Actual result:

Components like CRLSets error out

Expected result:

No error

Reproduces how often:

Easily

Brave version (brave://version info)

1.12.42 Chromium: 83.0.4103.106 (Official Build) nightly (64-bit)

mherrmann · 2020-11-10T09:17:35Z

Compiling some findings here.

Starting Brave Nightly with flags --enable-logging=stderr --v=1 on Linux produces the following output when the update errors happen:

[21534:21534:1110/101526.834284:VERBOSE1:component_updater_service.cc(324)] CheckForUpdates: automatic updatecheck for components.
[21563:21638:1110/101526.841124:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: https://go-updater.brave.com/extensions
[21534:21534:1110/101527.138161:VERBOSE1:request_sender.cc(181)] Request completed from url: https://update.googleapis.com/service/update2/json
[21534:21534:1110/101527.143297:VERBOSE1:component_updater_service.cc(411)] Update completed with error 0

Even though the log says https://update.googleapis.com/..., I don't think Brave actually requests this URL. Because with --vmodule=layer_tree_\*=0 --v=5, the next line is:

[21153:21153:1110/101137.266722:VERBOSE2:request_sender.cc(128)] Omaha response received: )]}'
{"response":{"protocol":"3.1","server":"prod","app":[{"appid":"gccbbckogglekeggclmmekihdgdpdgoe","status":"ok","updatecheck":{"status":"noupdate","urls":{"url":[{"codebase":"https://brave-core-ext.s3.brave.com/release/gccbbckogglekeggclmmekihdgdpdgoe/extension_1_0_207.crx"}]},"manifest":{"version":"1.0.207","packages":{"package":[{"name":"extension_1_0_207.crx","hash_sha256":"ea238e1705765fec151a7e5ebf8e985bac0b72c484420d3ff9d1aec03eef2e55","required":true}]}}}},{"appid":"oofiananboodjbbmdelgdommihjbkfag","status":"ok","updatecheck":{"status":"noupdate","urls":{"url":[{"codebase":"https://brave-core-ext.s3.brave.com/release/oofiananboodjbbmdelgdommihjbkfag/extension_1_0_18.crx"}]},"manifest":{"version":"1.0.18","packages":{"package":[{"name":"extension_1_0_18.crx","hash_sha256":"60bd9cd8e13bf6907541c269c7b0c5bad2c8739a089f957a81be8c85e0d029f7","required":true}]}}}},{"appid":"jicbkmdloagakknpihibphagfckhjdih","status":"ok","updatecheck":{"status":"noupdate","urls":{"url":[{"codebase":"https://brave-core-ext.s3.brave.com/release/jicbkmdloagakknpihibphagfckhjdih/extension_1_0_15.crx"}]},"manifest":{"version":"1.0.15","packages":{"package":[{"name":"extension_1_0_15.crx","hash_sha256":"ab93d08e7e06a89dcdad08ebe7bbe562177be9ee87337f1ec87d28cb35eb468e","required":true}]}}}},{"appid":"afalakplffnnnlkncjhbmahjfjhmlkal","status":"ok","updatecheck":{"status":"noupdate","urls":{"url":[{"codebase":"https://brave-core-ext.s3.brave.com/release/afalakplffnnnlkncjhbmahjfjhmlkal/extension_1_0_36.crx"}]},"manifest":{"version":"1.0.36","packages":{"package":[{"name":"extension_1_0_36.crx","hash_sha256":"585bf81a85b11809a8252b5575e6d9ec4d472eaa85db573b11f16ea2676e333d","required":true}]}}}},{"appid":"cffkpbalmllkdoenhmdmpbkajipdjfam","status":"ok","updatecheck":{"status":"noupdate","urls":{"url":[{"codebase":"https://brave-core-ext.s3.brave.com/release/cffkpbalmllkdoenhmdmpbkajipdjfam/extension_1_0_754.crx"}]},"manifest":{"version":"1.0.754","packages":{"package":[{"name":"extension_1_0_754.crx","hash_sha256":"60b2e40bbf0f0d755b6a7319ceaf2f9e8ce27bdeb9f82c2270591d1d33e85e29","required":true}]}}}}]}}

This seems to be a response from Brave servers.

The components whose automatic update produce "Update errors" for me are:

MEI Preload (appid aemomkdncapdnfajjbbcbdebjljbpmpj)
Legacy TLS Deprecation Configuration (appid bklopemakmnopmghhmccadeonafabnal)
Federated Learning of Cohorts
Autofill States Data
Subresource Filter Rules
Crowd Deny
Certificate Error Assistant
CRLSet
Safety Tips
File Type Policies
Origin Trials
Zxcvbn Data Dictionaries

I get "Component not updated" for the remaining 5 components:

Brave Local Data Updater
Brave Ad Block Updater
NTP Sponsored Images (US)
Brave SpeedReader Updater
Brave HTTPS Everywhere Updater

These successes / errors are the same across Brave restarts.

What's interesting is that error'd automatic update requests do succeed manually. So pressing the "Check for update" button next to "MEI Preload" for example works and eventually says "Component not updated".

My hypothesis is the following: During an automatic update, the client asks the server "I have components 1,2,3,... Is there an update?" I can see in the logs that the server responds with only a partial response. It says "there are no updates for components 3, 7, 8". So it does not say "there is no update for component 1". I believe it is this that leads to the update errors.

My hypothesis is further confirmed by the fact that checking for an update for a specific component manually does return the "no update" available response.

Specifically:

Update request for all components:

{"request":{"@os":"linux","@updater":"","acceptformat":"crx2,crx3","app":[
{"appid":"gccbbckogglekeggclmmekihdgdpdgoe","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"1.0.207"},
{"appid":"gcmjkmgdlgnkkcocmoeiminaijmmjnii","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"0.0.0.0"},
{"appid":"giekcmmlnklenlaomppkphknjmnnpneh","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"0.0.0.0"},
{"appid":"llkgjffcdpffmhiakmfcdcblohccpfmo","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"0.0.0.0"},
{"appid":"hfnkpimlhhgieaddgfemjhofmfblmnib","cohort":"1:jcl:","cohorthint":"Auto","cohortname":"Auto","enabled":true,"packages":{"package":[{"fp":"1.4546a79d9eb6aaf241f1c987a269dd39acf71e7aa1f572980a4b302fe5263f2e"}]},"ping":{"ping_freshness":"{f0d899d8-c88c-4045-8973-f245532c5686}","rd":5062},"updatecheck":{},"version":"6230"},
{"appid":"aemomkdncapdnfajjbbcbdebjljbpmpj","enabled":true,"ping":{"ping_freshness":"{af3f4db4-4e0a-449c-963f-47d1cdc91f31}","rd":5062},"updatecheck":{},"version":"1.0.5.0"},
{"appid":"jflookgnkcckhobaglndicnbbgbonegd","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"0.0.0.0"},
{"appid":"ggkkehgbnfjpeggfpleeakpidbkibbmn","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"0.0.0.0"},
{"appid":"cmahhnpholdijhjokonmfdjbfmklppij","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"0.0.0.0"},
{"appid":"khaoiebndkojlmppeemjhbpbandiljpe","cohort":"1:cux:","cohorthint":"Auto","cohortname":"Auto","enabled":true,"packages":{"package":[{"fp":"1.3aed9261364e0fba70475eca99d89ac4e9049c96ac6791e3213d0483f98bff6f"}]},"ping":{"ping_freshness":"{9ceca768-afb1-44f7-aee4-dc9a21ae0150}","rd":5062},"updatecheck":{},"version":"43"},
{"appid":"bklopemakmnopmghhmccadeonafabnal","cohort":"1:swl:","cohorthint":"Auto","enabled":true,"packages":{"package":[{"fp":"1.6dce22b9a11fa1e62b22559c4a157ce745e7fc63c6c6941a82cf11e8ecf65b0e"}]},"ping":{"ping_freshness":"{f5d2c4f3-a249-43e2-8fbf-de4e8e164807}","rd":5062},"updatecheck":{},"version":"3"},
{"appid":"ojhpjlocmbogdgmfpkhlaaeamibhnphh","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"0.0.0.0"},
{"appid":"eeigpngbgcognadeebkilcpcaedhellh","cohort":"1:w59:","cohorthint":"Auto","enabled":true,"packages":{"package":[{"fp":"1.c64c9c1008f3ba5f6e18b3ca524bc98dcd8acfae0a2720a8f1f3ef0f8d643d05"}]},"ping":{"ping_freshness":"{c6f05d40-c48b-4f34-bed2-29a12e287796}","rd":5062},"updatecheck":{},"version":"2020.11.2.164946"},
{"appid":"oofiananboodjbbmdelgdommihjbkfag","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"1.0.18"},
{"appid":"jicbkmdloagakknpihibphagfckhjdih","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"1.0.15"},
{"appid":"afalakplffnnnlkncjhbmahjfjhmlkal","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"1.0.36"},
{"appid":"cffkpbalmllkdoenhmdmpbkajipdjfam","enabled":true,"ping":{"r":-2},"updatecheck":{},"version":"1.0.754"}],
"arch":"x64","dedup":"cr","hw":{"physmemory":15},"lang":"","nacl_arch":"x86-64","os":{"arch":"x86_64","platform":"Linux","version":"5.8.0-0.bpo.2-amd64"},"prodchannel":"stable","prodversion":"87.1.18.38","protocol":"3.1","requestid":"{155ed67b-ed80-4bb8-a718-3ebbff216017}","sessionid":"{b0072630-bfa9-4910-a559-23f9593fba8f}","updaterchannel":"stable","updaterversion":"87.1.18.38"}}

Response - note that many of the components for which an update was requested are missing:

{"response":{"protocol":"3.1","server":"prod","app":[{
"appid":"gccbbckogglekeggclmmekihdgdpdgoe","status":"ok","updatecheck":{"status":"noupdate","urls":{"url":[{"codebase":"https://brave-core-ext.s3.brave.com/release/gccbbckogglekeggclmmekihdgdpdgoe/extension_1_0_207.crx"}]},"manifest":{"version":"1.0.207","packages":{"package":[{"name":"extension_1_0_207.crx","hash_sha256":"ea238e1705765fec151a7e5ebf8e985bac0b72c484420d3ff9d1aec03eef2e55","required":true}]}}}},{
"appid":"cffkpbalmllkdoenhmdmpbkajipdjfam","status":"ok","updatecheck":{"status":"noupdate","urls":{"url":[{"codebase":"https://brave-core-ext.s3.brave.com/release/cffkpbalmllkdoenhmdmpbkajipdjfam/extension_1_0_754.crx"}]},"manifest":{"version":"1.0.754","packages":{"package":[{"name":"extension_1_0_754.crx","hash_sha256":"60b2e40bbf0f0d755b6a7319ceaf2f9e8ce27bdeb9f82c2270591d1d33e85e29","required":true}]}}}},{
"appid":"oofiananboodjbbmdelgdommihjbkfag","status":"ok","updatecheck":{"status":"noupdate","urls":{"url":[{"codebase":"https://brave-core-ext.s3.brave.com/release/oofiananboodjbbmdelgdommihjbkfag/extension_1_0_18.crx"}]},"manifest":{"version":"1.0.18","packages":{"package":[{"name":"extension_1_0_18.crx","hash_sha256":"60bd9cd8e13bf6907541c269c7b0c5bad2c8739a089f957a81be8c85e0d029f7","required":true}]}}}},{
"appid":"jicbkmdloagakknpihibphagfckhjdih","status":"ok","updatecheck":{"status":"noupdate","urls":{"url":[{"codebase":"https://brave-core-ext.s3.brave.com/release/jicbkmdloagakknpihibphagfckhjdih/extension_1_0_15.crx"}]},"manifest":{"version":"1.0.15","packages":{"package":[{"name":"extension_1_0_15.crx","hash_sha256":"ab93d08e7e06a89dcdad08ebe7bbe562177be9ee87337f1ec87d28cb35eb468e","required":true}]}}}},{
"appid":"afalakplffnnnlkncjhbmahjfjhmlkal","status":"ok","updatecheck":{"status":"noupdate","urls":{"url":[{"codebase":"https://brave-core-ext.s3.brave.com/release/afalakplffnnnlkncjhbmahjfjhmlkal/extension_1_0_36.crx"}]},"manifest":{"version":"1.0.36","packages":{"package":[{"name":"extension_1_0_36.crx","hash_sha256":"585bf81a85b11809a8252b5575e6d9ec4d472eaa85db573b11f16ea2676e333d","required":true}]}}}}]}}

On the other hand, here is the manual update request for a failed component (MEI Preload):

{"request":{"@os":"linux","@updater":"","acceptformat":"crx2,crx3","app":[{"appid":"aemomkdncapdnfajjbbcbdebjljbpmpj","enabled":true,"installsource":"ondemand","ping":{"ping_freshness":"{af3f4db4-4e0a-449c-963f-47d1cdc91f31}","rd":5062},"updatecheck":{},"version":"1.0.5.0"}],"arch":"x64","dedup":"cr","hw":{"physmemory":15},"lang":"","nacl_arch":"x86-64","os":{"arch":"x86_64","platform":"Linux","version":"5.8.0-0.bpo.2-amd64"},"prodchannel":"stable","prodversion":"87.1.18.38","protocol":"3.1","requestid":"{634731f7-ab1c-4247-a437-fc05a529b054}","sessionid":"{6695c16e-6687-459e-a577-0797f803282a}","updaterchannel":"stable","updaterversion":"87.1.18.38"}}

This component was not mentioned in the response to the automatic update check above. But it is mentioned in the response to the specific request:

{"response":{"server":"prod","protocol":"3.1","daystart":{"elapsed_seconds":6346,"elapsed_days":5062},"app":[{"appid":"aemomkdncapdnfajjbbcbdebjljbpmpj","cohort":"","status":"ok","cohortname":"","ping":{"status":"ok"},"updatecheck":{"status":"noupdate"}}]}}

The hypothesis is also confirmed by the fact that Chrome's servers do return noupdate for all apps (/components) in the automatic update check. That is, if you ask https://update.googleapis.com/service/update2/json "I have components 1,2,3", then it returns (for example) "1 has no update, 2 has an update, 3 has no update". So Google's server implementation behaves differently from Brave's, and I think this is where the problem originates.

mherrmann · 2020-11-10T15:12:05Z

@mihaiplesa do you know who maintains https://go-updater.brave.com/extensions? I have a suspicion that the problem described in this issue is caused by a bug in that server implementation.

mihaiplesa · 2020-11-10T15:22:22Z

@jumde @linhkikuchi might be able to point in the right direction.

mherrmann · 2020-11-10T18:12:06Z

Hi @jumde @linhkikuchi I think https://go-updater.brave.com/extensions may be returning invalid responses in some cases. Specifically: Brave's component update check sends a request to this URL asking "I have components 1,2,3, is there an update?". But the URL only responds "there is no update for component 2". It should, I think, respond "there are no updates for components 1, 2, 3". Google's implementation https://update.googleapis.com/service/update2/json does this. Further details are above. Are you able to look into this, do you know who can look into it, or can you give me instructions how I can look into it?

jumde · 2020-11-11T06:45:41Z

@mherrmann - Code for go-updater.brave.com is here: https://github.com/brave/go-update. For google-components, go-updater redirects to componentupdater.brave.com which is a proxy to update.googleapis.com.

mherrmann · 2020-11-11T06:54:28Z

Thanks @jumde!

mherrmann · 2020-11-11T09:57:29Z

@jumde I looked at go-update and it seems to confirm my suspicion. What do you think of my hypothesis? Here it is again, in detail: I suspect that the update client expects to receive a noupdate response for all components it requests an update for. But go-update only returns noupdate for those components which are in DynamoDb. This is unlike Google's implementation, https://update.googleapis.com/service/update2/json, which always gives at least a noupdate for each component that was requested.

We could verify my hypothesis by you adding an entry to DynamoDb for one of the failing components listed above.

If my hypothesis turns out to be correct: Should I submit a PR to go-update that fetches and caches responses from componentupdater.brave.com?

mherrmann · 2020-11-13T06:28:11Z

@jumde an update please? I'm blocked on your response.

jumde · 2020-11-13T17:59:28Z

@mherrmann - If the component is not in the DynamoDB, the request is redirected to https://componentupdater.brave.com which is a proxy to https://update.googleapis.com. So the response should be noupdate.

Let me dig into this a bit more to see what's going on, will share an update shortly.

mherrmann · 2020-11-13T18:14:55Z

I think the request is only redirected when only one component is requested. It seems to me that when multiple components are requested, only those that are in Brave's DynamoDb are returned. That's where the problem comes from, I think.

Thank you for looking into it!

mherrmann · 2020-11-16T09:03:42Z

@jumde advised me via Slack how this can be fixed. He agrees with my analysis of the cause of the problem. However, my proposed solution won't work. For legal reasons, the Widevine component must be fetched from Google's servers. We are not allowed to set up a Brave proxy that mirrors Google's updates to this component.

Just to recapitulate: In an automatic update check, the component updater asks "are there updates for <brave component 1>, <brave component 2>, ..., <google component 1>, <google component 2>, ...". For legal reasons, Brave's update server can't return responses for all Google components, in particular the Widevine component. But the updater expects to receive a response for all the components it requested. This leads to the error described in this issue.

An important detail is that Brave's update server (go-updater) is able to handle update requests for a single Google component. The reason for this is that in this case it returns a redirect to Google's update server. This is what happens in on-demand updates of single components, hence why those updates do work.

To fix the problem, @jumde suggested we iterate over the components one by one. I'm looking into how best to implement this. Currently, I'm torn between component_updater_service, update_client or even deeper, such as by creating a Brave-specific update_checker_factory and a corresponding update_checker. A good entry point to inject our implementation could be the ComponentUpdateServiceFactory in component_updater_service.cc.

mherrmann · 2021-01-07T17:21:18Z

Verified that my fix works in Nightly v1.20.65 on Linux and Android.

Linux:

Android:

GeetaSarvadnya · 2021-01-21T16:10:30Z

Verification passed on


Brave | 1.20.86 Chromium: 88.0.4324.96 (Official Build) dev (64-bit)
-- | --
Revision | 68dba2d8a0b149a1d3afac56fa74648032bcf46b-refs/branch-heads/4324@{#1784}
OS | Windows 10 OS Version 2004 (Build 19041.746)

Verified STR from the description

Clean profile

Confirmed CRLSet is listed on brave://components as 6376 component updated on a clean profile.
Confirmed after ~1-minute reload component page is showing the component as up-to-date and same version is reflected.

Example	Example

Upgraded profile

Installed 1.19.78 (Dev) and confirmed CRLSet is listed on brave://components as 6376 Updated.
After ~1 minute component showed Update Error.
Updated to 1.20.86 (Dev).
Confirmed CRLSet is listed on brave://components as 6376 Component not updated.
Confirmed after ~1 minute reload component page is showing the component as up-to-date and same version is reflected.

Example 1.19.x	Example 1.19.x	Example 1.20.x	Example 1.20.x

Verification passed on

Brave	1.20.86 Chromium: 88.0.4324.96 (Official Build) dev (64-bit)
Revision	68dba2d8a0b149a1d3afac56fa74648032bcf46b-refs/branch-heads/4324@{#1784}
OS	Ubuntu 18.04 LTS

Verified test plan from the description

Clean profile
Verified CRLSet is listed on brave://components as 6376 component updated on a clean profile.
Verified after ~1-minute reload component page is showing the component as up-to-date and same version is reflected.

Upgraded profile. Upgraded from 1.19.x.
Verified after ~1 minute reload component page is showing the component as up-to-date and same version is reflected.

Verified passed with

Brave | 1.20.88 Chromium: 88.0.4324.96 (Official Build) dev (x86_64)
-- | --
Revision | 68dba2d8a0b149a1d3afac56fa74648032bcf46b-refs/branch-heads/4324@{#1784}
OS | macOS Version 10.15.7 (Build 19H15)

Confirmed no component errors.

Clean profile 1.20.x

Clean install 1.20.x	After ~8 min

Upgrade from 1.19.x to 1.20.x

Clean install 1.19.x	Upgrade to 1.20.x	After ~8 min

jumde added OS/Desktop feature/components labels Jun 25, 2020

rebron added the priority/P2 A bad problem. We might uplift this to the next planned release. label Aug 18, 2020

mherrmann self-assigned this Nov 10, 2020

mherrmann mentioned this issue Nov 17, 2020

Fix failing component auto-updates brave/brave-core#7164

Merged

23 tasks

This was referenced Nov 27, 2020

Fix crowd-deny updates #10280

Closed

Enable widevine cdm component updater in linux #7081

Closed

mherrmann changed the title ~~[Desktop] Google components on-demand update fails~~ [Desktop] Google components automatic update fails Dec 2, 2020

This was referenced Dec 2, 2020

Don't do explicit on demand component update check while startup #12983

Open

Remove unsupported components in brave://components #8709

Closed

mihaiplesa added this to the 1.20.x - Nightly milestone Dec 14, 2020

mihaiplesa added QA/Yes release-notes/include labels Dec 29, 2020

mihaiplesa closed this as completed in brave/brave-core#7164 Jan 7, 2021

GeetaSarvadnya added the QA Pass-Win64 label Jan 21, 2021

btlechowski added the QA Pass-Linux label Jan 21, 2021

LaurenWags added the QA Pass-macOS label Jan 25, 2021

rebron changed the title ~~[Desktop] Google components automatic update fails~~ Google components automatic update fails Jan 25, 2021

LaurenWags mentioned this issue Feb 8, 2021

[Desktop] Release Notes for 1.20.x Release #1 (Release Channel) #14035

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Google components automatic update fails #10464

Google components automatic update fails #10464

jumde commented Jun 25, 2020 •

edited by mherrmann

Loading

mherrmann commented Nov 10, 2020 •

edited

Loading

mherrmann commented Nov 10, 2020

mihaiplesa commented Nov 10, 2020

mherrmann commented Nov 10, 2020

jumde commented Nov 11, 2020 •

edited

Loading

mherrmann commented Nov 11, 2020

mherrmann commented Nov 11, 2020 •

edited

Loading

mherrmann commented Nov 13, 2020

jumde commented Nov 13, 2020

mherrmann commented Nov 13, 2020

mherrmann commented Nov 16, 2020 •

edited

Loading

mherrmann commented Jan 7, 2021

GeetaSarvadnya commented Jan 21, 2021 •

edited by LaurenWags

Loading

Google components automatic update fails #10464

Google components automatic update fails #10464

Comments

jumde commented Jun 25, 2020 • edited by mherrmann Loading

Description

Steps to Reproduce

Actual result:

Expected result:

Reproduces how often:

Brave version (brave://version info)

mherrmann commented Nov 10, 2020 • edited Loading

mherrmann commented Nov 10, 2020

mihaiplesa commented Nov 10, 2020

mherrmann commented Nov 10, 2020

jumde commented Nov 11, 2020 • edited Loading

mherrmann commented Nov 11, 2020

mherrmann commented Nov 11, 2020 • edited Loading

mherrmann commented Nov 13, 2020

jumde commented Nov 13, 2020

mherrmann commented Nov 13, 2020

mherrmann commented Nov 16, 2020 • edited Loading

mherrmann commented Jan 7, 2021

GeetaSarvadnya commented Jan 21, 2021 • edited by LaurenWags Loading

jumde commented Jun 25, 2020 •

edited by mherrmann

Loading

mherrmann commented Nov 10, 2020 •

edited

Loading

jumde commented Nov 11, 2020 •

edited

Loading

mherrmann commented Nov 11, 2020 •

edited

Loading

mherrmann commented Nov 16, 2020 •

edited

Loading

GeetaSarvadnya commented Jan 21, 2021 •

edited by LaurenWags

Loading