Observing consistently lower performance scores when running Lighthouse programmatically than from Chrome DevTools

[wolvriene]

Settings Used in both Programmatically and DevTool audit are same and are listed below .
Chrome Version : 78.0.3904.108
Node.js version: 13.1.0
Device :Desktop
Audits: Performance
Throttling : No Throttling(Provided by environment)
Other details provided by report:
User agent (host): Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/78.0.3904.108 Safari/537.36
User agent (network): Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3694.0 Safari/537.36 Chrome-Lighthouse

Ran audit on Devtool and by Programmatically .Results are different .
Programmatic Performance Score: 22
First Contentful Paint
1.7 s
Speed Index
4.0 s
Time to Interactive
6.5 s
First Meaningful Paint
4.4 s
First CPU Idle
6.5 s
Max Potential First Input Delay
280 ms

DevTools Performance Score: 82
Metrics
First Contentful Paint
0.7 s
Speed Index
2.8 s
Time to Interactive
5.0 s
First Meaningful Paint
5.0 s
First CPU Idle
5.0 s
Max Potential First Input Delay
480 ms
Please revert if any other additional information is required .

[patrickhulce]

How are you setting "no throttling" programmatically? Are you disabling the storage reset in DevTools?

[wolvriene]

Hi @patrickhulce
i have set "no throttling " programatically as below

       disableNetworkThrottling: true,
       disableCpuThrottling: true,
       throttlingMethod: 'provided',

Also disabled the storage reset in DevTools as "Clear storage"
attached in screenshots.
Please let me know for any changes

[patrickhulce]

Hm, that is how you set no throttling in programmatic so everything looks good there (just throttlingMethod: 'provided' should be all that's needed). The only other thing that seems to be different is headless vs. headful Chrome.

Is this URL public for us to reproduce the issue?

[wolvriene]

@patrickhulce
I checked with both headless and headful Chrome .
There is no much difference, Observing consistently lower performance scores even in headful (Programmatically)

URLs are not public

[wolvriene]

Hi @patrickhulce
Any update on this ?
Thanks

[patrickhulce]

Wait @wolvriene something is not right with the numbers you've reported. You shouldn't be getting anywhere near a 22 with those metric values.

In fact the difference in the metric values themselves is pretty well within the range of reasonable variance for different Chrome variants

[wolvriene]

@patrickhulce :
Please find a different screenshot with similar kind of number in which i am getting score as 22%.
I tried with lh-scorec-calc getting score 71%, is there any setting/configuration need to change ?

[patrickhulce]

Oooooh, is your version of programmatic newer than the DevTools version of Lighthouse?

Newer Lighthouse will automatically adjust the scores to desktop curves while older stable DevTools does not yet do this. This would explain the score discrepancy.

[wolvriene]

@patrickhulce
i am using Programmatic lighthouse version: 5.6.0

How to check lighthouse DevTool version ?
i am using Chrome Version : 78.0.3904.108

And, also which Performance score report is correct or trustable, DevTool or Programmatic ? As there is huge difference between them.

[patrickhulce]

The DevTools Lighthouse version can be found at the bottom of every report, Chrome 78 is 5.2.0.

The programmatic one in this case is more reflective of the performance because it contains the fix for adjusting score weights.

[wolvriene]

@patrickhulce

Lighthouse Version are same for DevTool and programmatic are same i.e. 5.6.0
Please find screenshot for the same.

But still there is huge difference Performance score between them.

Also, Could you please provide more specific about score weights and how it actually works.

[patrickhulce]

That's not the right report @wolvriene, we already know that programmatic is 5.6.0. The DevTools report could not possibly be produced by HeadlessChrome and if you're using Chrome 78, the Lighthouse version is 5.2.0.

Also, Could you please provide more specific about score weights and how it actually works.

There are different score curves for desktop and mobile, the mobile curve is more forgiving. When you test as a desktop device, in 5.6.0 it uses the desktop curve for the score. In older versions (i.e. 5.2.0), the desktop curve was not enforced properly and so the more forgiving curve was used instead which resulted in erroneously higher scores.

lighthouse/lighthouse-core/audits/metrics/first-contentful-paint.js

Lines 37 to 52 in 4c4f488

	static get defaultOptions() {
	return {
	mobile: {
	// 75th and 95th percentiles HTTPArchive -> median and PODR
	// https://bigquery.cloud.google.com/table/httparchive:lighthouse.2018_04_01_mobile?pli=1
	// see https://www.desmos.com/calculator/2t1ugwykrl
	scorePODR: 2000,
	scoreMedian: 4000,
	},
	desktop: {
	// SELECT QUANTILES(renderStart, 21) FROM [httparchive:summary_pages.2018_12_15_desktop] LIMIT 1000
	scorePODR: 800,
	scoreMedian: 1600,
	},
	};
	}

lighthouse/lighthouse-core/audits/metrics/first-contentful-paint.js

Lines 64 to 65 in 4c4f488

	const scoreOptions =
	context.options[artifacts.TestedAsMobileDevice === false ? 'desktop' : 'mobile'];

[wolvriene]

@patrickhulce

Thanks for the update
Yes in DevTool version is 5.2.0, Apologies for Incorrect screenshot, mistakenly i have uploaded it
I will more look into score weights and will close ticket shortly.

[wolvriene]

@patrickhulce

I have updated chrome and current Version is 79.0.3945.79, lighthouse version is 5.5.0

Does lighthouse version is 5.5.0 is having fix for adjusting score weights.

Also i Checked with Chrome Canary Version Version 81.0.3992.4, lighthouse version is 5.7.0.
Assuming it is having fix for adjusting score weights.But it does not have throttling option.

Is there any way i can compare programmatic lighthouse results with DevTools lighthouse results and conclude the same.

As there is huge difference between the performance score between them.

[patrickhulce]

Is there any way i can compare programmatic lighthouse results with DevTools lighthouse results and conclude the same.

tl;dr - Yes but not with the settings you have.

First of all, to be clear, the metric results themselves should still be comparable between your two reports, it's just the scoring curve we're talking about here. The only versions of Lighthouse in DevTools that support the score curve adjustments have also removed the throttling setting, so you must use simulated throttling from the command line to see similar scores.

FWIW, that specific commit I'm referring to might not have reached DevTools yet either, so it might take a bit of time to reach Canary.

[wolvriene]

@patrickhulce

Please correct me if i am wrong, so only Performance score will be affected by score curve, not other metrics for light house such as First Contentful Paint , speed index , Time to Interactive etc.

[patrickhulce]

Performance score will be affected by score curve, not other metrics for light house such as First Contentful Paint , speed index , Time to Interactive etc.

Correct, the raw metric values are unaffected by the score curve. The score curve is only for the performance score. (Note that the text color of the metrics might be different based on the score curve, but the value itself should not be affected)