Accuracy of CLI tool vs one that's on the developer tools on chrome

[aprilmintacpineda]

Provide the steps to reproduce

1. Run LH on a URL using developer tools on chrome.
2. Run LH on the same URL using CLI available on NPM.

What is the current behavior?

Results are drastically different.

• I got 91 on performance which I was not expecting because using the developer tools on chrome I got only 66.
• Using dev tools on chrome, I got TTFB = ERROR!. Using the CLI, I didn't.

The only difference is with the performance. The rest are the same.

What is the expected behavior?

LH should show accurate performance measurement using CLI and Dev tool.

Environment Information

• Affected Channels: CLI, DevTools
• Lighthouse version: On my dev tool, it says 3.0.3. CLI is 4.0.0-alpha.2-3.2.1.
• Node.js version: v11.2.0
• Operating System: Mac OS

[patrickhulce]

We're going to need a little more information here. You're using two different versions of Lighthouse with unknown settings on an unknown URL. Do you think you could help fill in the blanks a bit?

[aprilmintacpineda]

About the difference in version, I did not install lighthouse on my devtool. I actually just saw it there. Should have I updated it? how do I update it?

About the url part, you can use https://stackoverflow.com/ as url, it doesn't really matter. I just tested it now on my computer using https://stackoverflow.com/ as the url. I suggest not logged in.

This image is the lighthouse on devtools.

This image is the lighthouse on CLI using Lighthouse Report Viewer.

[patrickhulce]

Thanks for sharing that @aprilmintacpineda! Was there something obscuring your Chrome window for some of the time in the CLI case? All of my runs (DevTools, Extension, and CLI) are similar to your DevTools run and not your CLI run.

There are many reasons why metrics can vary. We've documented the most common factors, but not every page load will be identical.

[aprilmintacpineda]

I don't think there is.

Differences in performance in each run is understandable since network speed is also a factor and its speed fluctuates. But the differences shouldn't be that big, right?

I don't know why the SEO are also different. Using CLI it shows 80 while on DevTool it shows 70.

The color coding are also different. Accessibility at 79 shows green on DevTool. Accessibility on CLI shows orange on the Viewer. This could just be the viewer itself.

Also, in the case above it shows my CLI is slower than the DevTool. On the site that I am developing, which I'm sorry but you can't access since it's exclusive to the office network. The performance is low on DevTool but and high on CLI. This is consistent.

This is using DevTool.

This is using CLI.

Result on measuring the performance is also very consistent with only 0 - 5 points differences each run.

On the site that I am developing, the only test that is different is the performance, I'm not sure why on the other sites, even SEO is different.

[aprilmintacpineda]

BTW.

Here's the options I use for DevTool

On CLI, I only do lighthouse <link> --output json --output-path test.json.

[patrickhulce]

@aprilmintacpineda applied throttling in some cases can be quite different from simulated throttling, the more direct comparison on the CLI would be using --throttling-method=devtools.

We're always trying to close the gap between them, but certain characteristics of sites can make it difficult to accurately simulate.

As for the SEO differences, those are different versions of LH so if they have different # audits then the score will be different for the same # of failures. If you find a particular audit to be flaky, then we'd love a separate bug report about it with repro steps if possible.

[aprilmintacpineda]

How do I update the lighthouse devtool to its latest version? I thought it's suppose to automatically update since my chrome's auto update is enabled.

I just checked for update. My chrome is up-to-date Version 70.0.3538.110 (Official Build) (64-bit)

[patrickhulce]

@aprilmintacpineda it does update automatically, but it runs behind the CLI by quite a bit to go through Chrome's release process. Current latest version of LH will likely hit Chrome stable around January-February.

[aprilmintacpineda]

applied throttling in some cases can be quite different from simulated throttling, the more direct comparison on the CLI would be using --throttling-method=devtools.

You we're right. I tried running my devtool using Simulated throttling and it gave me the same range, 91-93 on performance. Why isn't there an option to do Applied throttling using the CLI?

When I did --throttling-method=devtools it gave me even lower scores on Performance and PWA was down because the site doesn't load fast enough.

[patrickhulce]

Applied throttling is --throttling-method=devtools. The differences between CLI and DevTools are just down to Chrome profile and any potential extensions you might have (an adblocker or antivirus extension perhaps?)

[aprilmintacpineda]

I don't have those kinds of extensions. The CLI also opens chrome browser and navigates to the given link. So I assume it's using the browser's environment as well, similar to running lighthouse on the DevTool, yes?

I checked out lighthouse chrome extension. It's version is the same as the one I have on my DevTool.