☔ Pagespeed Insights results differ from lighthouse in chrome

[vertic4l]

Hey there!

I'm optimizing a mobile site and lighthouse reported me a score of 92. So far so good i thought. But after checking back with Pagespeed Insights, which also uses lighthouse, i'm getting a score of 57.

Is there any reliable way to get the same score?

[exterkamp]

Hey, thanks for reaching out!

To help with diagnosing this we'll need to have the URL you're testing, and the context that you are running Lighthouse in.

How did you get the 92? Are you running on Devtools in Chrome? On the node CLI? And what throttling settings are you using?

[vertic4l]

Hey @exterkamp,

i've made an audit with lighthouse in DevTools in Chrome and now tested it with node CLI aswell.
Lighthouse in DevTools gives a score of 92 (Performance) right now. Lighthouse (latest, 4.0.0-alpha.2-3.2.1) reports 75 for Best Practices and 0 for Performance. But still huge difference to the PSI score. Mailed you the corresponding website!

These are my settings:

Last test with lighthouse in DevTools:

Last test with PSI:

Last test with CLI

[exterkamp]

So I ran that URL against all channels and I am finding things to be a consistently in the high 50's low 60's for performance. I would say within expected variance.
Node v4.0.0-alpha.1

PSI (I also force ran against an EU data center, and got similar results)

DevTools in Production Chrome v3.0.3

The 0 for performance in the CLI, that definitely seems like an error, that site shouldn't get a 0, and if it consistently does without surfacing an error that might be a bug or something might be blocking LH from running locally?

If you run from other machines does your devtools still get such a high score? That seems unreasonably high given those settings, I am seeing a consistent ~10 seconds for TTI, but that screenshot of devtools running is ~4 seconds, seems oddly fast compared to the other runs.

[vertic4l]

@exterkamp Thanks for testing it. It's odd that there's no Performance score when using the CLI version (although Best Practices gets a score as you can see). And it's very odd that the TTI differs so much.

[vertic4l]

@exterkamp So, i made some changes to pass some more of the audits for mobile.

Lighthouse in DevTools
Passed audits: 15
Performance Score: 83

Pagespeed Insights
Passed Audits: 12
Performance Score: 45 (sometimes 55)

Both tests were made on newly updated production site.

[AlexVadkovskiy]

Currently on Chromium Version 71.0.3578.80 I get the same weird results for almost any website. Lighthouse in DevTools is always 25-45 points better than on PSI.
example: https://www.omgubuntu.co.uk/ (got 82 in lighthouse and around 50 on PSI)

Could this be related to DNS lookups or similar stuff? Could PSI server have a little bit slower access to the tested website than my local machine (depending on where the server actually located in US or in EU?

[vertic4l]

@AlexVadkovskiy just checked the website (https://www.omgubuntu.co.uk/) from Germany.

PSI: 29
Lighthouse in DevTools: 90

(mobile scores)

[exterkamp]

Hmmm checking out https://www.omgubuntu.co.uk/ I get:
~80 from cli on v4.0.0-alpha.1
~40 from pagespeed insights (running v4.0.0-alpha.1)

So this is solidly reproducible for that site.

I am force running pagespeed in the EU and seeing similar results. I will say that pagespeed has a consistent ~200ms TTFB while running locally have ~60ms TTFB, and that URL seems to send more images to pagespeed for some reason? Some odd behavior for sure, and some questionable latency which might be us or the site.

@patrickhulce might want to take a look at this from a lantern perspective on why this could be different/maybe something is up with the trace.

@paulirish might want to take a look at this from a global pagespeed latency perspective.

[patrickhulce]

@exterkamp is this the same case as @wardpeet brought up? some of his runs in LR had every request duplicated which would immediately explain 2x lantern predictions.

@AlexVadkovskiy the Chromium difference is also probably explained by #6772 FYI

[exterkamp]

@patrickhulce I remember that, but I can't find that issue/discussion, you have a link to it/can you bump it to mention this thread?

[patrickhulce]

but I can't find that issue/discussion, you have a link to it/can you bump it to mention this thread?

IIRC that's because we just discussed it over chat :) @wardpeet do you happen to still have those?

[vertic4l]

@exterkamp any update here?

[vertic4l]

@patrickhulce so there is a bug in chrome which is mentioned in other issues. Will this fix the PSI score as well ?

[patrickhulce]

@vertic4l to which bug are you referring?

[exterkamp]

Oops this got buried! Missed that bump. I will definitely be looking into these URLs again soon. So far we have:

• https://www.omgubuntu.co.uk/
• @wardpeet other url?

Taking a guess, the bug might be ignoring flags? But that shouldn't be an issue because this was reported before that version of Chrome shipped iirc.

[ashtonlance]

I'm having the complete opposite experience with this. My dev tools Lighthouse is give me a score of 37 where PSI is giving a score of 91.

The url in question is: https://biketours.com

[patrickhulce]

@ashtonlance I'm seeing PSI score of 24 for that URL

[ashtonlance]

@patrickhulce Sorry, those number I gave were the desktop scores.

[patrickhulce]

@ashtonlance Ah you're probably experiencing #6772 then. Give it a whirl in Chrome Canary.

[ashtonlance]

Lighthouse is busted for me in Canary Version 73.0.3672.0.

I was able to get the Lighthouse CLI to produce similar results as PSI by turning off emulation and throttling.

[amaladevi-r]

@exterkamp I'm seeing the same issue, where Lighthouse reports a high score while PSI shows a lower one. You can try on this url

https://www.bankbazaar.com/credit-card.html?mobileSite=true

[vertic4l]

Well, not that bad...

[patrickhulce]

@ashtonlance something is off, that's Lighthouse 3.0.0-beta.0 which is ~8 months old, not the right Canary version 🤔

[ashtonlance]

@patrickhulce Weird indeed. That was from a build I downloaded at https://www.google.com/chrome/canary/. However, I just downloaded a fresh copy and all seems to be good now.

[vertic4l]

@exterkamp any news?

[hooligani]

PSI is not run with a real browser (only trying to simulate browsers to cut infrastructure costs) so that's why the results differ from Lighthouse in chrome (observed mode).

[matti]

And it is almost impossible to run PSI locally, see https://twitter.com/patrickhulce/status/1331961018654855168?s=20

So just don't use PSI, use gtmetrix.com for one user and one page or supervisor.com for multiple pages and multiple users

[socialpreneur]

@h3d0

Can we somehow get one single reliable source of metric? Which one does Google use to rank the website pages?

Google has made clear they don't use lab data for ranking but field data, so basically the scores are nothing as it used to be. But in order to optimize and fix problems, lab data is the only thing we can immediately see the changes (field data requires 28 days), so we still have to use it as one of the tools to measure performance along with webpagetest and others.

[Sven74Muc]

Sorry, I missed this thread... have the same issue: #11908

[pjar]

Since joining this thread I have stopped messing with Pagespeed and am ignoring Core Web Vitals and life is good. Not an option for everyone, I know, but other than actually streamlining my site/s as much as possible and eliminating any bottlenecks (which you want to do anyway) there is no point, seemingly, in obsessing about Google's metrics because they make no sense.

I have same feelings. I'm spending my weekend tasked with improving this laughable metrics. With every click on 'analyze' on page insights I'm getting scores diffing by about 20 points, listing different 'reasons' each time. How am I supposed to use it?

I went to check how their page https://developers.google.com/web/tools/lighthouse/ that is supposed to tell us how we should prepare our pages to please this vague tool does not do well for mobile score - barely 40 points and it is as well -+10 points after each 'analyze'. Looks like even google can't figure out how to please this tool.

Lots of negatives here, but I'm just trying to finish my work and spend some time with family instead of trying to guess what google/lighthouse wants from me.

[VenkatSandeeph]

I have run PSI and Lighthouse in Chrome DevTools for mobile. Found the FCP and LCP in both cases are varying. Also, i can find the different results when checked from different locations. Once try on this url https://mydepartments.in/.

[matti]

@VenkatSandeeph yeah, you site loads way faster than what those screenshots show

[amtechnet]

I am trying to optimize my site and getting the same issue with this link- https://cardinsider.com/best-credit-cards-in-india/

[natew]

I'm getting drastically different numbers for http://tamagui.dev

Locally mobile gets 95 performance, pagespeed.web.dev gives 46, consistently for each.

Both show Moto with slow 4G. The only difference I see is pagespeed runs headless. The big difference is Total Blocking Time, which to me indicates that React Concurrent Suspense hydration isn't running properly. I can turn of Suspense locally and I get numbers much closer to pagespeed. But it's just a guess.

[connorjclark]

@natew I'm getting around ~PSI's score when I run locally.

Do you happen to have Clear Storage enabled?

To debug further, it'd great if you could do the same Lighthouse run with the CLI and -GA options enabled, then send us the latest_run artifacts folder. That may help determine why we get different results.

[natew]

Locally running lighthouse https://tamagui.dev -GA I get 🤷‍♂️ (ran it twice and got the same). I'm on a hotspot but the entire test ran in 10 seconds or so so the reported time of 34s makes no sense.

[connorjclark]

We simulate the timings. https://github.com/GoogleChrome/lighthouse/blob/master/docs/throttling.md So unless you're running with non-default settings to apply no network emulation, you can't compare the results with how long it took in real time.

Although... being on a hotspot may impact the input/quality of the simulation. Not certain, but I would be weary of that being an accurate test.

[natew]

Got a 20 again at home.

latest-run.zip

This is what I get in browser:

Edit: that is running incognito and not, cleared data. Tested the same on Brave.

[eliawk]

same problem here, when using Chrome dev Tools I got better results than Pagespeed insights. So i don't know how test a local website with consistent values.

[gilbertococchi]

@eliawk please make sure you apply "DevTools Throttling (advanced)" when testing Lighthouse in Chrome DevTools, that should help to match a similar emulation with the PSI one.

[eliawk]

@gilbertococchi thanks for your answer, but I was already using "DevTools Throttling (advanced)". Even with that setting I get 95 on Chrome and 67 on Pagespeed as you can see from the screenshots:

[gilbertococchi]

I suspect TBT times are different, perhaps because of CPU condition, Extensions blocking 3Ps or other stuff.
I would suggest testing using an Incognito Tab, Total Blocking Times should be more similar but there is always a level of variability with TBT due to the different way 3Ps may react trace by trace.

More info here: https://developers.google.com/web/tools/lighthouse/variability

[Hibrix-net]

I suspect TBT times are different, perhaps because of CPU condition, Extensions blocking 3Ps or other stuff. I would suggest testing using an Incognito Tab, Total Blocking Times should be more similar but there is always a level of variability with TBT due to the different way 3Ps may react trace by trace.

More info here: https://developers.google.com/web/tools/lighthouse/variability

I agree, you have to adjust throttling and CPU conditions and perform it in an incognito window. Both DevTools Lighthouse and PSI are lab testings and both rely to different conditions. The local Lighthouse relies in your specific PC settings simulation and the PSI relies on it own simulation. Don't expect to have exact same values in both testings anyway.

[Jelenbl]

really extreme difference to same location

Mobile
PSI - 35
Light - 96

both going to https://dev.d33869rf57mkjo.amplifyapp.com/
Light used in incognito tab and non with dev tools advanced on

chrome version: 118.0.5993.70

[Hibrix-net]

really extreme difference to same location

Mobile PSI - 35 Light - 96

both going to https://dev.d33869rf57mkjo.amplifyapp.com/ Light used in incognito tab and non with dev tools advanced on

chrome version: 118.0.5993.70

I recommend verifying your server's performance from the USA, specifically from Mountain View, California. This is crucial because PageSpeed Insights (PSI) conducts its requests from that location. By doing so, you can get a more accurate representation of how your site performs in PSI tests.

[suraj-sundariya]

HI @adamraine/ @connorjclark/@exterkamp, I'm inquiring about the progress on this issue. It has been open since 2018, and there hasn't been any update regarding ongoing work or an ETA of completion. Many people/companies rely on PSI and Lighthouse, so could you please share an update?

[matti]

There is no way to properly solve this: PSI machines are running some cpu architecture in some data center in somewhere - unless Lighthouse is running in the same machine, same network and same time, the results will vary.

Also, it makes no sense to just test for one page - for example my startup, https://www.supervisor.com provides 24/7 automatic web vital monitoring with real browsers that are running with dedicated resources to provide consistency in the results.

Once a browser clicks automatically the entire site and checks the metrics, it's natural to extend this as a load test also by just adding more of the same browsers (something that we also do)