Different Lighthouse Performance Results from the same computer

[tleish]

FAQ

• Yes, my issue is not about variability or throttling.
• Yes, my issue is not about a specific accessibility audit (file with axe-core instead).

URL

https://www.google.com/

What happened?

I'm getting different performance results between Google Chrome > Dev Tools > Lighthouse and lighthouse-cli from the same computer. lighthouse-cli seems to always show much worse performance than in dev tools. My assumption is the defaults of the browser and cli are the same, but I'm guessing I'm wrong and unclear how to align these to be the same.

What parameters do I need to set in order to get the two results similar.

What did you expect?

I expected the performance results to be close to the same between browser and cli tools.

What have you tried?

Chrome DevTools Steps

1. Open google chrome in incognito mode (to avoid interference of plugins etc).
2. Navigate to https://www.google.com/
3. Open Chrome Dev Tools > Lighthouse
4. Check Performance (uncheck other metrics)
5. Select Desktop
6. Click “Analyze page Load”

Chrome DevTools Results

CLI Steps

lighthouse https://www.google.com --preset=desktop --only-categories=performance

CLI Results

How were you running Lighthouse?

CLI, Chrome DevTools

Lighthouse Version

9.6.8

Chrome Version

109.0.5414.87 (shows Lighthouse v9.6.8 in results)

Node Version

18.6.0

OS

macOS 13.0.1

[adamraine]

I can't reproduce the ~42 performance score you're seeing on the CLI, are you able to share the report file? Also, please read our variability docs.

[tleish]

I understand the variability, but a "Time to interactive" difference of 0.3s to 4.7s from and overall performance score of around half on the same machine (with lots of CPU and RAM) doesn't make any sense to me. To me it almost seems as if the CLI tool is counting the time it takes to start chrome as part of the performance.

I'm attaching the report file.

www.google.com_2023-01-17_16-09-40.report.html.zip

I've tested this from the same machine on two different networks with the same results. I also had a co-worker run the same exercise (Chrome DevTools and CLI) on their Macbook and they experience the same results as me.

[adamraine]

Can you run this command:

lighthouse https://www.google.com --preset=desktop --only-categories=performance -GA

Then send the contents of the resulting ./latest-run folder.

[tleish]

www.google.com_2023-01-19_13-56-51.zip

[tleish]

Testing this again several weeks, the results are now as I would now expect.

[Chandan-TY]

Hi @tleish
Could you let me know what was the solution for the above issue, I'm facing exactly the same issue. In MAC OS there is a huge difference in Performance percentage.
Total Blocking time is more when compare to dev tools and running on MAC cli.

[tleish]

Hi @tleish Could you let me know what was the solution for the above issue

I'm not sure what changed. It just started working as expected one day.

[TigerC10]

I'm currently experiencing this myself. Lighthouse testing in the browser shows performance score of 97, but CLI testing shows a performance score of 34.

Every online tester I'm using, including https://pagespeed.web.dev/ agrees with the CLI testing. But there's something crazy happening here because CLI and the online tests show a FCP > 5 seconds. But in my actual browser I'm getting <1.5 seconds total load until visually complete. The CLI is reporting wildly inaccurate measurements for the actual human experience.

And before you say "browser cache" let me just clarify that the pages load instantly on first load - it's not like first load takes >5 seconds and then my chrome dev tools lighthouse report hits cached data.

I don't know why the CLI testing is so wildly off.

[adamraine]

Every online tester I'm using, including pagespeed.web.dev agrees with the CLI testing. But there's something crazy happening here because CLI and the online tests show a FCP > 5 seconds. But in my actual browser I'm getting <1.5 seconds total load until visually complete. The CLI is reporting wildly inaccurate measurements for the actual human experience.

https://github.com/GoogleChrome/lighthouse#why-is-the-performance-score-so-low-it-looks-fine-to-me

Locking this thread because it is old. If you have further issues please file a new bug report.