Inaccurate and inconsistent results in Google App Engine

[tostaylo]

I am receiving inaccurate, inconsistent results when running Lighthouse performance audits while deployed from Google App Engine. Not sure if it is Lighthouse or Puppeteer which isn't playing nice in the Google App Engine environment.
Below is the function I am calling.

const lighthouse = require('lighthouse')
const puppeteer = require('puppeteer')
async function launchPuppeteerRunLighthouse(url) {
  try {
    const browser = await puppeteer.launch({
      args: [
        '--no-sandbox',
        '--incognito',
        '--disable-gpu',
        '--disable-dev-shm-usage',
        '--disable-setuid-sandbox',
        '--no-first-run',
        '--no-zygote',
      ],
    })
    const port = browser._connection._url.slice(15, 20)
    const { lhr, report } = await lighthouse(url, {
      port,
      output: 'html',
      onlyCategories: ['performance'],
      logLevel: 'debug',
    })
    browser.close();
    return { report, lhr }
  } catch (error) {
    console.log(error)
  }
}

Here is my app.yaml

runtime: nodejs8
instance_class: F4_1G

When I run a performance audit for https://www.google.com programmatically on my local development server I am observing results consistent with running the performance audit using Lighthouse located in Chrome Dev Tools.
When running the performance audit for https://www.google.com programmatically on Google App Engine I am receiving much lower performance scores.
I have tested in both environments with dozens of audits. I am attaching screenshots for what I would consider an average result of a performance audit of https://www.google.com in both environments.

[patrickhulce]

The report you got from app engine and my gut about the instance types used in app engine point to the same culprit: woefully underpowered CPU capacity.

In the app engine report, while loading google.com there was 8 seconds of CPU time spent just executing JavaScript, which is crazy. My assumption is that app engine uses the equivalent of burst-able instances, which means you might get 2.4GHz execution, but only for a brief period before the CPU is throttled to a crawl. Running Chrome and loading webpages is not an easy task for underpowered hardware. Our own testing infrastructure showed high variability until moving up to at least n1-standard-2 machines (2 dedicated, non-burstable cores and 8GB RAM).

App engine/lambda/etc will get the job done for screenshots/test flows/crawling but when measuring performance, you really need beefier hardware unfortunately.

cc @benschwarz @denar90 in case they have any insight to share with running LH on cloud hardware.

[benschwarz]

I'd echo the same sentiments @patrickhulce. Using underpowered hardware will only lead to inconsistent results.

[tostaylo]

Thank you @patrickhulce and @benschwarz for the quick responses. I had a feeling hardware was going to be the issue. Do you have any suggestions on cloud solutions capable of running Chrome and Lighthouse efficiently? Or am I out of luck?

[patrickhulce]

@tostaylo n1-standard-2 standard machines work well for our testing infrastructure (each runs a web server, Chrome, and Lighthouse locally), but if you're looking for a function-as-a-servicee/autoscaling environment, I'm not aware of any that will give you reliable performance results.

[denar90]

@patrickhulce @tostaylo we also faced the same kind of problems at @treosh. After moving to stable dedicated instance we increased results stability as well. Check out the public report for google.com https://treo.sh/demo/4/pages/5?interval=1year
btw, @alekseykulikov is champion in that question 🥇

[tostaylo]

After testing on a Google Compute Engine n1-standard-2 instance, I am observing Lighthouse performance audits consistent with what I observe while testing in my local Chrome. Thank you @patrickhulce and @benschwarz for the recommendation.

[tostaylo]

@denar90 Thanks for showing me @treosh. Really good looking site! I'm all ears for anything you learned about running Lighthouse on a VM instance.

[sunildarji811]

@tostaylo How do you use lighthouse with Google App Engine Node js Env? I am getting error like "the environment variable chrome_path must be set to executable of a build of chromium version 54.0 or later".

Any way to install Chromium on App Engine?

Can you share your idea on above issue?