Cannot bundle Lighthouse with user flows

[sam-barker]

We had been trying to utilise fraggle rock so we can have lighthouse flows run in a lambda function.

However the way that we build the code is using typescript and webpack, and this minimises everything into one file.

Because of this, the lambda explodes as it cannot find the json files references within shared/localization/locales, as the code is as follows:

const files = {
  'ar': JSON.parse(fs.readFileSync(`${__dirname}/locales/ar.json`, 'utf8')),
  'ar-XB': JSON.parse(fs.readFileSync(`${__dirname}/locales/ar-XB.json`, 'utf8')),
  'bg': JSON.parse(fs.readFileSync(`${__dirname}/locales/bg.json`, 'utf8')),
  'ca': JSON.parse(fs.readFileSync(`${__dirname}/locales/ca.json`, 'utf8')),
  'cs': JSON.parse(fs.readFileSync(`${__dirname}/locales/cs.json`, 'utf8')),
  'da': JSON.parse(fs.readFileSync(`${__dirname}/locales/da.json`, 'utf8')),
  'de': JSON.parse(fs.readFileSync(`${__dirname}/locales/de.json`, 'utf8')),
  'el': JSON.parse(fs.readFileSync(`${__dirname}/locales/el.json`, 'utf8')),
  'en-GB': JSON.parse(fs.readFileSync(`${__dirname}/locales/en-GB.json`, 'utf8')),
  'en-US': JSON.parse(fs.readFileSync(`${__dirname}/locales/en-US.json`, 'utf8')),
  'en-XA': JSON.parse(fs.readFileSync(`${__dirname}/locales/en-XA.json`, 'utf8')),
  'en-XL': JSON.parse(fs.readFileSync(`${__dirname}/locales/en-XL.json`, 'utf8')),
  ... etc

Our lambda files consist of:

/var/task/
  - index.js

Of course naturally when it tries to resolve ${__dirname}/locales/en-XL.json this won't work, as the JSON files aren't in our lambda.

I'm not quite sure why it was decided to include json files like this? Wouldn't it make more sense just to require or import them directly instead of this weird dirname way of doing it? That would stop our lambda from blowing up!

[adamraine]

If you aren't interested in non-default locales, a workaround is to shim locales.js with an empty module. We do this in a couple places with rollup and rollup-plugin-shim:

lighthouse/build/build-viewer.js

Lines 45 to 47 in abb88b3

	rollupPlugins.shim({
	'./locales.js': 'export default {}',
	}),

IDK the proper way to do this in webpack, null-loader maybe?

I'm not quite sure why it was decided to include json files like this? Wouldn't it make more sense just to require or import them directly instead of this weird dirname way of doing it? That would stop our lambda from blowing up!

It was done in anticipation of converting to ESM: #12721

[sam-barker]

@adamraine thanks we can get around this. However we can't get around

const libDetectorSource = fs.readFileSync(
  require.resolve('js-library-detector/library/libraries.js'), 'utf8');

Being used in lighthouse-core/gather/gatherers/stacks.js. This also makes the lambda blow up as obviously we don't have this file to be able to read. Are there any workarounds for this?

[adamraine]

We use a custom rollup plugin to inline our fs calls for bundles. Updates on that are tracked in #13231.

We used to use https://www.npmjs.com/package/brfs, perhaps a webpack solution with https://www.npmjs.com/package/transform-loader could work?

Solving this solution with libraries.js may just reveal more issues. Ultimately I think we just don't support bundling like this.

---

You might have some success forking https://github.com/GoogleChrome/lighthouse/blob/master/build/build-bundle.js to compile Lighthouse assets. Example:

node ./build/build-bundle.js entry.js bundle.js

entry.js could be:

module.exports = require('./lighthouse-core/fraggle-rock/api.js');

Perhaps we could expose build-bundle.js in the NPM package to support something.

[paulirish]

Aye. we would recommend deferring to us for bundling... We have enough of these special cases it's going to be hard for you to maintain a bundling setup that works with our codebase.

The last suggestion from adam is as close to our preferred approach. We just havent yet needed to make this bundle, but we certainly could.

[connorjclark]

re: #13398 (comment)

Have you tried installing js-library-detector? There are surely a handful of dependencies needed that we only define in our devDependencies, which would result in them missing.

Unfortunately in the JS ecosystem it is sometimes necessary to checkout a project manually and tweak its build for complex use cases. You can try patch-package to assist in that.

---

We currently publish a few bundles, but they are only for report code. We could bundle a new entry using ./lighthouse-core/fraggle-rock/api.js, but it would be pretty big, so maybe as a separate npm package?

Publishing build/build-bundle.js would run into quite a few issues re: packages defined in our devDependencies being missing/not matching in the consuming project.

[paulirish]

@sam-barker take a look at https://github.com/GoogleChrome/lighthouse/compare/buildfr

if you checkout that branch. you can run build-flow-bundle. it'll create a dist/flow-bundle.js which i presume should run in lambda. (our bundles are designed for running in webworkers or browser pages.)

the API is exposed on globalThis.fr

from there, i suppose you can work with that.. but as you probably already know.. you'll need to be passing in a puppeteer page etc.