Lighthouse performance result significantly different when run on CLI vs Dev Tools #14810

tkane2000 · 2023-02-18T03:38:58Z

Hi!
I'm having trouble getting consistent results when I run Lighthouse on CLI vs Dev Tools on the same site.

I've run most tests on against our dev server, but I ran against nytimes.com so we can test against a public facing site and got the same results.

The performance score for CI is always significantly lower.

Settings

Here's the command I'm running:
lighthouse "https://www.nytimes.com/" --output json --preset desktop

The settings I'm using in devtools are:

legacy navigation: unchecked
clear storage: checked
Simulated throttling (default)
mode: navigation
Device: Desktop
Categories: Peformance

Browser versions:
chrome-debug: Version 110.0.5481.100 (Official Build) (x86_64 translated)
The one I'm running devtools on: Version 110.0.5481.100 (Official Build) (arm64)

I've disabled all extensions in my browser.
When the test runs the browser window is visible and I don't touch anything until the test finishes.

Lighthouse version for both browsers: 9.6.8
Throttling seems to be exactly the same (from looking at the "Custom Throttling" link in the footer of the results

Results

CLI

Performance: 26
First Contentful Paint: 0.5 s
Time to Interactive: 11.2 s
Speed Index: 8.7 s
Total Blocking Time: 8,100 ms
Largest Contentful Paint: 6.7 s
Cumulative Layout Shift: 0.005

devtools

Performance: 63
First Contentful Paint: 0.8 s
Time to Interactive: 4.8 s
Speed Index: 2.6 s
Total Blocking Time: 160 ms
Largest Contentful Paint: 4.0 s
Cumulative Layout Shift: 0.03

Note the huge difference in blocking time. I notice that the page is slower.

Full results for 2 passes here:
lighthouse-reports.zip

I've also played around with various settings on CI and the results are always similar to CLI. This doesn't seem to be an issue with variance because the results for devtools is consistent as are the result for CI and CLI. If it was variance, I'd expect them all to fluctuate more randomly.

Here are some of the settings I tried in CI:

module.exports = {
  ci: {
    collect: {
      startServerCommand: 'npm run benchmark-server',
      headful: true,
      // ...
      settings: {
        onlyCategories: ['performance'],
        preset: 'desktop',
        screenEmulation: {
          width: 1367
        },
        /** Throttlng */
        // throttlingMethod: 'devtools',
        throttlingMethod: 'simulate',
        throttling: {
          requestLatencyMs: 0, 
          downloadThroughputKbps: 0, 
          uploadThroughputKbps: 0,
          cpuSlowdownMultiplier: 1,
        }
      }
    },
    assert: {
      assertions: {
        'categories:performance': ['warn', { minScore: 1 }],
      },
    },
    upload: {
      target: 'temporary-public-storage',
    },
  },
};

Lastly, I realize that nytimes.com might not be the best site to test against because it's content can change and it runs ads, etc, but I tested it at different times and got the same result. Also, I've ran the comparison 30 times or more against our dev site and the content there doesn't change at all. It must be something with my local machine, but my goal is to get CI set up on Jenkins and I need to have this working first.

Please let me know if you need any more information or if there's anything else I can try here!

Related Issues:

The text was updated successfully, but these errors were encountered:

connorjclark · 2023-02-18T07:14:00Z

Thanks for the detailed report!

I can respond more fully after the weekend, but in the meantime it'd be helpful if you could repeat your analysis using 10.0.

you can grab the latest from npm for the CLI, and use Chrome Canary to use 10.0 in DevTools.

It's a huge amount of changes between the two versions, so good to rule out that we haven't already fixed whatever this particular problem is first.

connorjclark · 2023-02-18T07:18:06Z

My first observation is that you've used legacy navigation for the CLI (that is the default for 9.x, in 10 it is not), but used the new navigation runner in CDT. It shouldn't matter, but it could be exposing an issue between the two.

The chrome that the CLI opens.. can you open a new tab in it for chrome://version/ and share the version you see? Could be #14288 again somehow (are you on Mac?)

tkane2000 · 2023-02-19T01:38:21Z

Thanks for the suggestions. Unfortunately, I’ll be unexpectedly without my laptop until Friday. :/ Sorry for the inconvenience! I will try your suggestions on a different laptop tomorrow though.On Feb 18, 2023, at 2:18 AM, Connor Clark ***@***.***> wrote: My first observation is that you've used legacy navigation for the CLI (that is the default for 9.x, in 10 it is not), but used the new navigation runner in CDT. It shouldn't matter, but it could be exposing an issue between the two. The chrome that the CLI opens.. can you open a new tab in it for chrome://about and share the version you see? Could be #14288 ... —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

tkane2000 · 2023-02-23T04:46:40Z

are you on a mac?

Yes

can you open a new tab in it for chrome://version/ and share the version you see?

Google Chrome | 110.0.5481.177 (Official Build) (x86_64 translated)
-- | --
Revision | f34f7ab2d4ca4ad498ef42aeba4f4eb2c1392d63-refs/branch-heads/5481@{#1239}
OS | macOS Version 12.6.2 (Build 21G320)
JavaScript | V8 11.0.226.16
User Agent | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36

running CLI v10 returns the following:

$ lighthouse "https://www.nytimes.com/" --output html --preset desktop --legacy-navigation false
Runtime error encountered: Launching Chrome on Mac Silicon (arm64) from an x64 Node installation results in Rosetta translating the Chrome binary, even if Chrome is already arm64. This would result in huge performance issues. To resolve this, you must run Lighthouse CLI with a version of Node built for arm64. You should also confirm that your Chrome install says arm64 in chrome://version
Error: Launching Chrome on Mac Silicon (arm64) from an x64 Node installation results in Rosetta translating the Chrome binary, even if Chrome is already arm64. This would result in huge performance issues. To resolve this, you must run Lighthouse CLI with a version of Node built for arm64. You should also confirm that your Chrome install says arm64 in chrome://version
    at getDebuggableChrome (file:///Users/foo/.nvm/versions/node/v14.16.1/lib/node_modules/lighthouse/cli/run.js:77:13)
    at runLighthouse (file:///Users/foo/.nvm/versions/node/v14.16.1/lib/node_modules/lighthouse/cli/run.js:237:30)
    at begin (file:///Users/foo/.nvm/versions/node/v14.16.1/lib/node_modules/lighthouse/cli/bin.js:135:10)
    at async file:///Users/foo/.nvm/versions/node/v14.16.1/lib/node_modules/lighthouse/cli/index.js:10:1

So, looks like you're guess was correct, this is the same issue as #14288

I'll try to install the correct version of node and reply here with results.

tkane2000 · 2023-02-24T14:42:27Z

I upgraded node to v15.14.0 and set CHROME_PATH=/Applications/Google\ Chrome.app so that lighthouse uses that binary, which is: 110.0.5481.177 (Official Build) (arm64), but I still get "Runtime error encountered: Launching Chrome on Mac Silicon (arm64) from an x64 Node installation...". :\

Update: I also tried:
$ CHROME_PATH=~/Applications/Google\ Chrome\ Canary.app

Any idea what I'm missing here?

Dospios · 2023-02-28T21:51:21Z

I have the same performance issue between devTools and CLI/node versions. On all platforms, on v10.0.1 and on v9.6.8 with --legacy-navigation false.

tkane2000 · 2023-03-08T21:19:25Z

Ok, I fixed the issue and was able to get the arm64 versions of node and lighthouse working and the results are much more consistent! All had a performance score of around 91 except for a couple from devtools which were in the high 80s...still pretty close and can probably be attributed to variances on nytimes.com, like ads.

Reports (4 from each):
lighthouse-reports.zip

Just in case anyone runs into similar issue, here's how I got it running:

Use the latest version of node (as you can see, I'm using nvm to switch versions)
$ nvm ls-remote v16 # find the latest
$ nvm i v16.19.1
$ nvm use v16.19.1
$ node -v
v16.19.1

Use Chrome Canary
Set lighthouse to use Chrome Canary because it has the latest version of lighthouse and support arm64.
If you have extensions installed in Chrome Canary, make sure they're all be disabled.

$ export CHROME_PATH=~/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary
(you can see in my previous comment that I was initially using the wrong path)
you can also put this in your ~/.zshrc file

Open via command line
~/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary
and check in browser tab:
chrome://version/
113.0.5639.0 (Official Build) canary (arm64)

Install the latest lighthouse globally
$ npm i -g lighthouse

In my case, for some reason, it still tries to use the version of lighthouse in the previous node version:
$ which lighthouse
/Users/myusername/.nvm/versions/node/v15.14.0/bin/lighthouse
as a workaround I run it with the full path, like this:
$ /Users/myusername/.nvm/versions/node/v16.19.1/bin/lighthouse ...

Run lighthouse cli
$ /Users/myusername/.nvm/versions/node/v16.19.1/bin/lighthouse "https://www.nytimes.com/" --output html --preset desktop --legacy-navigation false

Then, to run devtools, I opened chrome via cmd line like this:
$ ~/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary
(it probably isn't necessary to open it via cmd line...I was just trying to mimic lighthouse cli as much as possible)

Lastly, open dev tools and run lighthouse

sebwoolford · 2023-03-31T18:00:43Z

Hey, thanks for sharing, I'm hoping this will fix it for me as I'm encountering the same issues and am also on a Mac. I noticed a small typo and thought I'd let you know.

$ nvm ls-remove v16 # find the latest

Should be ls-remote not ls-remove

sebwoolford · 2023-03-31T18:20:01Z

Actually after digging deeper I think what fixed this issue for me is that, prompted by noticing you were adding --output html --preset desktop, I figured out that Lighthouse defaults the throttling to this: throttling: throttling.mobileSlow4G and I was expecting desktop-broadband-like results, ie. not really throttled.

I added desktop throttling to where I was invoking lighthouse like this:

const lighthouse = require('lighthouse');
const constants = require('lighthouse/lighthouse-core/config/constants');

const report = await lighthouse(url, {
	port: (new URL(browser.wsEndpoint())).port,
	throttling: constants.throttling.desktopDense4G
});

KingPrithviraj · 2024-10-21T04:56:48Z

hey guys am facing the same issue as am getting difference in lighthouse report while running by dev tool and cmd can anyone point out the solution

Dospios · 2024-10-21T06:58:18Z

hey guys am facing the same issue as am getting difference in lighthouse report while running by dev tool and cmd can anyone point out the solution

I have not solved this problem, but I have found a better solution. I use the pagespeed insights api and all such runs are done by the google robot instead of me, it's even better than the original version because based on the same metrics, google ranks pages in search and I get data from google itself

devtools-bot added the needs-priority label Feb 18, 2023

connorjclark closed this as completed Mar 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lighthouse performance result significantly different when run on CLI vs Dev Tools #14810

Lighthouse performance result significantly different when run on CLI vs Dev Tools #14810

tkane2000 commented Feb 18, 2023 •

edited

Loading

connorjclark commented Feb 18, 2023

connorjclark commented Feb 18, 2023 •

edited

Loading

tkane2000 commented Feb 19, 2023 via email

tkane2000 commented Feb 23, 2023 •

edited

Loading

tkane2000 commented Feb 24, 2023 •

edited

Loading

Dospios commented Feb 28, 2023

tkane2000 commented Mar 8, 2023 •

edited

Loading

sebwoolford commented Mar 31, 2023

sebwoolford commented Mar 31, 2023

KingPrithviraj commented Oct 21, 2024

Dospios commented Oct 21, 2024

Lighthouse performance result significantly different when run on CLI vs Dev Tools #14810

Lighthouse performance result significantly different when run on CLI vs Dev Tools #14810

Comments

tkane2000 commented Feb 18, 2023 • edited Loading

Settings

Results

Related Issues:

connorjclark commented Feb 18, 2023

connorjclark commented Feb 18, 2023 • edited Loading

tkane2000 commented Feb 19, 2023 via email

tkane2000 commented Feb 23, 2023 • edited Loading

tkane2000 commented Feb 24, 2023 • edited Loading

Dospios commented Feb 28, 2023

tkane2000 commented Mar 8, 2023 • edited Loading

sebwoolford commented Mar 31, 2023

sebwoolford commented Mar 31, 2023

KingPrithviraj commented Oct 21, 2024

Dospios commented Oct 21, 2024

tkane2000 commented Feb 18, 2023 •

edited

Loading

connorjclark commented Feb 18, 2023 •

edited

Loading

tkane2000 commented Feb 23, 2023 •

edited

Loading

tkane2000 commented Feb 24, 2023 •

edited

Loading

tkane2000 commented Mar 8, 2023 •

edited

Loading