Can't reuse chrome-canary instance between lighthouse runs

[calebeby]

FAQ

• Yes, my issue is not about variability or throttling.
• Yes, my issue is not about a specific accessibility audit (file with axe-core instead).

URL

https://github.com (irrelevant, happens with any url)

What happened?

I am not sure if this is a bug, or something that was never supposed to work, previously did, but doesn't now.

This is something that worked previously (Dec. 2022) and doesn't work now. I did not change the lighthouse version (it is and was 9.6.8), but it is possible/likely that chrome-canary updated since then.

I would like to reuse the chrome-canary instance between runs of lighthouse on different pages.

import chromeLauncher from 'chrome-launcher';
import lighthouse from 'lighthouse';
const chrome = await chromeLauncher.launch({
  chromeFlags: ['--headless', '--no-first-run'],
});

await lighthouse('https://github.com', {
  port: chrome.port,
});
await lighthouse('https://github.com', {
  port: chrome.port,
});

What did you expect?

I expect it to run lighthouse on the first page, then run lighthouse on the 2nd page.

Here is what happens (this error message appears after running the first lighthouse report; during the 2nd lighthouse call:

/Users/calebeby/Projects/lighthouse-parade/node_modules/lighthouse/lighthouse-core/gather/connections/cri.js:45
            return Promise.reject(new Error('Cannot create new tab, and no tabs already open.'));
                                  ^

Error: Cannot create new tab, and no tabs already open.
    at /Users/calebeby/Projects/lighthouse-parade/node_modules/lighthouse/lighthouse-core/gather/connections/cri.js:45:35
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Driver.connect (/Users/calebeby/Projects/lighthouse-parade/node_modules/lighthouse/lighthouse-core/gather/driver.js:119:5)
    at async GatherRunner.run (/Users/calebeby/Projects/lighthouse-parade/node_modules/lighthouse/lighthouse-core/gather/gather-runner.js:484:7)
    at async Runner._gatherArtifactsFromBrowser (/Users/calebeby/Projects/lighthouse-parade/node_modules/lighthouse/lighthouse-core/runner.js:264:23)
    at async Runner.gather (/Users/calebeby/Projects/lighthouse-parade/node_modules/lighthouse/lighthouse-core/runner.js:170:21)
    at async lighthouse (/Users/calebeby/Projects/lighthouse-parade/node_modules/lighthouse/lighthouse-core/index.js:50:21)
    at async file:///Users/calebeby/Projects/lighthouse-parade/example-bug-repro.mjs:11:1

I see that @paulirish has removed the code that displays this error message, but it looks like that change hasn't been released:
https://github.com/GoogleChrome/lighthouse/pull/14012/files

If I apply Paul's change to that file in my node_modules (commenting out the catch clause), this is the error I get instead:

/Users/calebeby/Projects/lighthouse-parade/node_modules/lighthouse/lighthouse-core/gather/connections/cri.js:107
          reject(new Error(`Protocol JSON API error (${command}), status: ${response.statusCode}`));
                 ^

Error: Protocol JSON API error (new), status: 405
    at IncomingMessage.<anonymous> (/Users/calebeby/Projects/lighthouse-parade/node_modules/lighthouse/lighthouse-core/gather/connections/cri.js:107:18)
    at IncomingMessage.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1344:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Can you clarify whether it is intended to be possible to reuse a chrome instance between lighthouse runs like this, and if not, can the error message be updated to be more clear?

What have you tried?

No response

How were you running Lighthouse?

node

Lighthouse Version

9.6.8

Chrome Version

112.0.5566.0 (Official Build) canary (x86_64)

Node Version

Node.js v18.3.0

OS

Mac

Relevant log output

No response

[calebeby]

Also, if anyone has some extra time to explain, would someone mind explaining to me what Fraggle Rock is? I don't see it defined here, and although I see it referenced everywhere, I have a hard time understanding what it is.

[adamraine]

TLDR try out a dev release which should make this work.

Can you clarify whether it is intended to be possible to reuse a chrome instance between lighthouse runs like this, and if not, can the error message be updated to be more clear?

I don't personally know the exact reasoning but I imagine it was a design feature that just stuck because Lighthouse was only interested in testing a single navigation in isolation (@paulirish is this correct?). In any case, Lighthouse has evolved a lot which brings me to your second question...

Also, if anyone has some extra time to explain, would someone mind explaining to me what Fraggle Rock is?

It started off as a mostly internal project name for adding user flow support (#11313), but it also includes several changes to the classic Lighthouse API that are relevant for your use case.

One of these changes to the classic API was that we would create a new tab in the browser for every Lighthouse run, which should make your use case possible. This change hasn't been officially released yet but you can test it out in one of our dev releases.