Build system refactor #8140

kumavis · 2020-02-28T12:42:54Z

status: awaiting review

this is primarily a change to the "build/task system" and not the "bundling system"

change summary

build time performance improvements!
new colorful build timeline output 🌈
manifest platform overrides now specified as json files, located in app/manifest/*.json
renamed entry tasks to [prod, dev, testDev, test]. yarn script names unchanged for now.
dropped image compression from build (super slow)
build system broken into parts in development/build/*.js, gulpfile is gone
dropped gulp for task composition for simple async functions
dropped gulp for entry points. now just uses a simple js script to read entry from process.argv
renamed "copy" tasks -> "static"

plz QA: your usual workflow

kumavis · 2020-02-28T12:48:53Z

added some candy

i find this timeline very helpful in identifying potential meaningful optimizations
eg even if you improve the speed of something, might not improve the time-to-completion bc everything is running parallel. but! its all running in a single node process so may still increase perf

metamaskbot · 2020-03-02T04:04:13Z

Builds ready [4ff258e]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
dep viz: background
all artifacts

Page Load Metrics (598 ± 61 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Notification	firstPaint	34	87	47	16	8
		domContentLoaded	316	812	596	127	61
		load	318	813	598	127	61
		domInteractive	316	811	596	127	61

kumavis · 2020-03-02T04:20:15Z

experimenting with running each bundle in separate processes
results are local, may differ in CI due to processor count
bout a 2x speedup

before (single)

prod 155.0s
scripts:deps:background 112.9s
scripts:deps:ui 84.5s
scripts:core:prod:inpage 26.8s
scripts:core:prod:background 84.8s
scripts:core:prod:ui 151.7s
scripts:core:prod:phishing-detect 41.3s
scripts:core:prod:contentscript 2.4s

after (separate)

prod 75.4s
scripts:deps:background 70.3s
scripts:deps:ui 42.0s
scripts:core:prod:inpage 12.6s
scripts:core:prod:background 70.1s
scripts:core:prod:ui 70.1s
scripts:core:prod:phishing-detect 12.3s
scripts:core:prod:contentscript 4.5s

kumavis · 2020-03-02T04:45:30Z

on CI prep-build 3m25s -> 2m29s

package.json

development/build/task.js

Co-Authored-By: Mark Stacey <[email protected]>

metamaskbot · 2020-03-08T02:57:25Z

Builds ready [a1bdc8a]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
dep viz: background
all artifacts

Page Load Metrics (680 ± 34 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Notification	firstPaint	36	108	61	27	13
		domContentLoaded	585	847	679	72	34
		load	587	849	680	72	34
		domInteractive	585	847	678	72	34

package.json

Gudahtt · 2020-03-08T03:34:15Z

I'm seeing this when a .scss file changes during a watch build:

(node:28142) UnhandledPromiseRejectionWarning: TypeError: stream.on is not a function
    at eos (/home/gudahtt/projects/metamask-extension/node_modules/end-of-stream/index.js:73:9)
    at eos (/home/gudahtt/projects/metamask-extension/node_modules/end-of-stream/index.js:14:41)
    at P (/home/gudahtt/projects/metamask-extension/node_modules/pify/index.js:49:6)
    at new Promise (<anonymous>)
    at /home/gudahtt/projects/metamask-extension/node_modules/pify/index.js:11:9
    at ret (/home/gudahtt/projects/metamask-extension/node_modules/pify/index.js:72:32)
    at watch (/home/gudahtt/projects/metamask-extension/development/build/styles.js:54:17)
    at write (/home/gudahtt/projects/metamask-extension/node_modules/gulp-watch/index.js:148:3)
    at /home/gudahtt/projects/metamask-extension/node_modules/gulp-watch/index.js:131:5

development/build/styles.js

Gudahtt · 2020-03-08T03:42:28Z

I think this is ready to go after those last three changes. I've tested it out locally (with the above suggested changes) and it all seems to work

Co-Authored-By: Mark Stacey <[email protected]>

kumavis · 2020-03-08T05:27:07Z

great catches, thanks

metamaskbot · 2020-03-08T05:45:19Z

Builds ready [132758f]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
dep viz: background
all artifacts

Page Load Metrics (643 ± 49 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Notification	firstPaint	35	92	43	12	6
		domContentLoaded	355	765	641	103	49
		load	356	767	643	102	49
		domInteractive	355	765	641	103	49

Gudahtt

LGTM!

The version bump script referenced the old file path for the manifest. It was stored as a single file called `manifest.json`, but it was split into `_base.json` and platform-specific manifests in #8140. The manifest version bump script has been updated to reference `_base.json`, which is the piece that has the version property.

This package has not been used since #8140. We now spawn separate processes directly in our build script rather than using this gulp plugin to do so.

This package hasn't been used since #8140, which dropped it for being too slow and of minimal benefit. We should consider re-adding this as a CI check to ensure images are optimized, but I don't think it should be re-added to the build process itself.

This package has not been used since #8140. We now spawn separate processes directly in our build script rather than using this gulp plugin to do so.

This package hasn't been used since #8140, which dropped it for being too slow and of minimal benefit. We should consider re-adding this as a CI check to ensure images are optimized, but I don't think it should be re-added to the build process itself.

kumavis added 11 commits February 27, 2020 11:42

build - start static asset task cleanup

f1be8af

build - simplify manifest tasks

9f5ee68

build - refactor + rename some tasks

b3c336b

build - various cleanups

2bae542

manifest - fix ref from controller

51895b8

build - drop gulp for simple async tasks

04a934d

build - breakout gulpfile into multiple files

e575c98

build - rename some tasks

0328465

build - use task fn refs instead of string names

fddb209

build - bundle all scripts first, except for contentscript

7fce940

build - improve task timeline

4a042a3

kumavis requested review from danjm, Gudahtt and whymarrh as code owners February 28, 2020 12:42

deps - update lock

d242e0a

kumavis added 9 commits March 2, 2020 09:33

build - improve task time printout

41b44f0

build/scripts - remove intermediate named task

cd39d38

build - use 'yarn build' for task entry points

5f82b00

build - properly run tasks via runTask for timeline display

e3fb072

development/announcer - fix manifest path + clean

36bd9a5

build - lint fix

78c8141

build - make all defined tasks possible entry points

83f5098

build/task - properly report errors during task

393263d

ci - fix sesify/lavamoat-viz build command

4ff258e

build/scripts - run each bundle in separate processes

32c399a

lint fix

609203d

kumavis mentioned this pull request Mar 7, 2020

Bundler Refactor #8170

Closed

Gudahtt reviewed Mar 7, 2020

View reviewed changes

package.json Outdated Show resolved Hide resolved

development/build/task.js Outdated Show resolved Hide resolved

kumavis and others added 2 commits March 8, 2020 10:37

Update package.json

bfa086f

Co-Authored-By: Mark Stacey <[email protected]>

build/task - remove unused fn

a1bdc8a

Gudahtt requested changes Mar 8, 2020

View reviewed changes

package.json Outdated Show resolved Hide resolved

Gudahtt requested changes Mar 8, 2020

View reviewed changes

development/build/styles.js Outdated Show resolved Hide resolved

development/build/styles.js Outdated Show resolved Hide resolved

kumavis and others added 3 commits March 8, 2020 13:25

Update package.json

a9dd12e

Co-Authored-By: Mark Stacey <[email protected]>

Update development/build/styles.js

9de374f

Co-Authored-By: Mark Stacey <[email protected]>

Update development/build/styles.js

132758f

Co-Authored-By: Mark Stacey <[email protected]>

Gudahtt approved these changes Mar 8, 2020

View reviewed changes

kumavis merged commit 7686eda into develop Mar 9, 2020

whymarrh deleted the build-sys-1 branch March 9, 2020 17:40

Gudahtt mentioned this pull request Jun 1, 2020

Fix manifest version bump script #8715

Merged

Gudahtt added a commit that referenced this pull request Feb 12, 2021

Remove gulp-multi-process

27a9fd5

This package has not been used since #8140. We now spawn separate processes directly in our build script rather than using this gulp plugin to do so.

Gudahtt mentioned this pull request Feb 12, 2021

Remove gulp-multi-process #10434

Merged

Gudahtt mentioned this pull request Feb 12, 2021

Remove gulp-imagemin #10435

Merged

Gudahtt added a commit that referenced this pull request Feb 12, 2021

Remove gulp-multi-process (#10434)

906324c

This package has not been used since #8140. We now spawn separate processes directly in our build script rather than using this gulp plugin to do so.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build system refactor #8140

Build system refactor #8140

kumavis commented Feb 28, 2020 •

edited

Loading

kumavis commented Feb 28, 2020 •

edited

Loading

metamaskbot commented Mar 2, 2020

kumavis commented Mar 2, 2020 •

edited

Loading

kumavis commented Mar 2, 2020 •

edited

Loading

metamaskbot commented Mar 8, 2020

Gudahtt commented Mar 8, 2020 •

edited

Loading

Gudahtt commented Mar 8, 2020

kumavis commented Mar 8, 2020

metamaskbot commented Mar 8, 2020

Gudahtt left a comment

Build system refactor #8140

Build system refactor #8140

Conversation

kumavis commented Feb 28, 2020 • edited Loading

kumavis commented Feb 28, 2020 • edited Loading

metamaskbot commented Mar 2, 2020

kumavis commented Mar 2, 2020 • edited Loading

kumavis commented Mar 2, 2020 • edited Loading

metamaskbot commented Mar 8, 2020

Gudahtt commented Mar 8, 2020 • edited Loading

Gudahtt commented Mar 8, 2020

kumavis commented Mar 8, 2020

metamaskbot commented Mar 8, 2020

Gudahtt left a comment

Choose a reason for hiding this comment

kumavis commented Feb 28, 2020 •

edited

Loading

kumavis commented Feb 28, 2020 •

edited

Loading

kumavis commented Mar 2, 2020 •

edited

Loading

kumavis commented Mar 2, 2020 •

edited

Loading

Gudahtt commented Mar 8, 2020 •

edited

Loading