misc(compare-runs): report metrics, add gather/audit modes, etc #10296

paulirish · 2020-02-04T21:59:32Z

Some upgrades to the collect-timings script!

Some new features:

`--gather` and `--audit` modes, and visual progress bar

had to save all artifacts into timings-data/ as well.

metrics are reported

Coefficient of variation is reported

i prefer this over reading the stddev in a big way.

other

rename --measure-filter to --filter
strict yargs

Split out from #9920

connorjclark · 2020-02-04T23:11:29Z

lighthouse-core/scripts/compare-timings.js

@@ -23,28 +25,32 @@ const argv = yargs
    // common flags
    'name': 'Unique identifier, makes the folder for storing LHRs. Not a path',
    'report-exclude': 'Regex of properties to exclude. Set to "none" to disable default',
-    // --collect
-    'collect': 'Saves LHRs to disk',
+    // collection


Suggested change

// collection

// --gather

can you group the params for the gather stage right after gather?

connorjclark · 2020-02-04T23:15:04Z

lighthouse-core/scripts/compare-timings.js

+      mkdirp.sync(gatherDir);
+      // TODO include multiple urls too.
+      const progressBar = new Array(Math.round(i * 40/ argv.n)).fill('▄').join('').padEnd(40);
+      console.log(`Gathering:   `, '|', progressBar, '|')


this would be so much better if you grabbed the logger from the lantern collect scripts (handles clearing the linefeed for you)

connorjclark · 2020-02-04T23:16:16Z

lighthouse-core/scripts/compare-timings.js

+        'node',
+        `${LH_ROOT}/lighthouse-cli`,
+        url,
+        `--audit-mode=${gatherDir}`,


i like being verbose when possible.

funny b.c. I had to look this up :)

connorjclark · 2020-02-04T23:18:11Z

lighthouse-core/scripts/compare-timings.js

-  if (argv.collect) collect();
+  if (argv.gather) gather();
+  if (argv.audit) audit();
+  if (argv.collect) {


backwards compatibility.

--collect did the full run before. now it does the same thing.

happy to do whatever tho.

connorjclark · 2020-02-04T23:19:08Z

lighthouse-core/scripts/compare-timings.js

@@ -23,28 +25,32 @@ const argv = yargs
    // common flags
    'name': 'Unique identifier, makes the folder for storing LHRs. Not a path',
    'report-exclude': 'Regex of properties to exclude. Set to "none" to disable default',
-    // --collect
-    'collect': 'Saves LHRs to disk',
+    // collection


can you group the params for the gather stage right after gather?

connorjclark · 2020-02-04T23:19:29Z

lighthouse-core/scripts/compare-timings.js

+    // collection
+    'gather': 'Just gathers',
+    'audit': 'Audits from the artifacts on disk',
+    'collect': 'Gathers, audits and saves LHRs to disk',


move this to under common flags

connorjclark · 2020-02-04T23:23:58Z

lighthouse-core/scripts/compare-timings.js

@@ -188,22 +238,24 @@ function filter(results) {
 }

 /**
- * @param {number=} value
+ * @param {number|string|undefined} value


why was string added here? it makes the type guard not make sense. maybe change body to typeof value === 'number'

mostly because i wanted to render CV as a percentage instead of a 0.XXX number.

but i did another solution which is better.

i do like using typeof number tho. did that.

connorjclark · 2020-02-04T23:24:09Z

lighthouse-core/scripts/compare-timings.js

-      const durations = durationsByName[entry.name] = durationsByName[entry.name] || [];
-      durations.push(entry.duration);
+      const durations = durationsByName[name] = durationsByName[name] || [];
+      durations.push(Number(timimg));


ah thx. renaming to duration anyway.

connorjclark · 2020-02-04T23:29:06Z

lighthouse-core/scripts/compare-timings.js

-  const results = aggregateResults(argv.name);
-  filter(results);
-  print(results);
+  for (const resultType of ['timings', 'metrics']) { 


not a fan of printing both of these unconditionally. seems you'd only want to look at timings or metrics, not both.

k done and documented.

connorjclark · 2020-02-04T23:33:16Z

seems a rename is in order. just collect?

paulirish

seem a rename is in order.

renamed to compare-runs.js

paulirish · 2020-02-05T00:54:52Z

lighthouse-core/scripts/compare-timings.js

+      mkdirp.sync(gatherDir);
+      // TODO include multiple urls too.
+      const progressBar = new Array(Math.round(i * 40/ argv.n)).fill('▄').join('').padEnd(40);
+      console.log(`Gathering:   `, '|', progressBar, '|')


paulirish · 2020-02-05T00:54:59Z

lighthouse-core/scripts/compare-timings.js

-      const durations = durationsByName[entry.name] = durationsByName[entry.name] || [];
-      durations.push(entry.duration);
+      const durations = durationsByName[name] = durationsByName[name] || [];
+      durations.push(Number(timimg));


ah thx. renaming to duration anyway.

paulirish · 2020-02-05T00:55:16Z

lighthouse-core/scripts/compare-timings.js

-  if (argv.collect) collect();
+  if (argv.gather) gather();
+  if (argv.audit) audit();
+  if (argv.collect) {


backwards compatibility.

--collect did the full run before. now it does the same thing.

happy to do whatever tho.

paulirish · 2020-02-05T00:56:25Z

lighthouse-core/scripts/compare-timings.js

+        'node',
+        `${LH_ROOT}/lighthouse-cli`,
+        url,
+        `--audit-mode=${gatherDir}`,


i like being verbose when possible.

paulirish · 2020-02-05T00:57:31Z

lighthouse-core/scripts/compare-timings.js

@@ -23,28 +25,32 @@ const argv = yargs
    // common flags
    'name': 'Unique identifier, makes the folder for storing LHRs. Not a path',
    'report-exclude': 'Regex of properties to exclude. Set to "none" to disable default',
-    // --collect
-    'collect': 'Saves LHRs to disk',
+    // collection


paulirish · 2020-02-05T00:58:25Z

lighthouse-core/scripts/compare-timings.js

@@ -188,22 +238,24 @@ function filter(results) {
 }

 /**
- * @param {number=} value
+ * @param {number|string|undefined} value


mostly because i wanted to render CV as a percentage instead of a 0.XXX number.

but i did another solution which is better.

paulirish · 2020-02-05T00:59:40Z

lighthouse-core/scripts/compare-timings.js

@@ -188,22 +238,24 @@ function filter(results) {
 }

 /**
- * @param {number=} value
+ * @param {number|string|undefined} value


i do like using typeof number tho. did that.

paulirish · 2020-02-05T01:42:13Z

lighthouse-core/scripts/compare-timings.js

-  const results = aggregateResults(argv.name);
-  filter(results);
-  print(results);
+  for (const resultType of ['timings', 'metrics']) { 


k done and documented.

connorjclark · 2020-02-05T02:05:34Z

lighthouse-core/scripts/compare-runs.js

+
+      // Push the aggregate time of each unique (by name) entry.
+      for (const [name, durationsForSingleRun] of Object.entries(durationsByName)) {
+        const key = `${lhr.requestedUrl}@@@${kind}@@@${name}`;


paulirish added 7 commits February 4, 2020 13:35

apply original comparetimings hacks

de62b98

generalizing

f74ea4f

cv

abe5a71

rename measure-filter to filter

2f6238d

types

2179b40

filter out timestamps from metrics

40e68ea

name

98c008a

paulirish requested a review from connorjclark February 4, 2020 21:59

paulirish requested a review from a team as a code owner February 4, 2020 21:59

googlebot added the cla: yes label Feb 4, 2020

devtools-bot added the needs-priority label Feb 4, 2020

connorjclark reviewed Feb 4, 2020

View reviewed changes

connorjclark requested changes Feb 4, 2020

View reviewed changes

paulirish added 6 commits February 4, 2020 16:49

cool progress logger

2bba172

flags

5345e5f

isNumber tweaks

8b55aef

get all metrics and entries and include that in the key

2ef6f62

move includeFilter implementation next to reportExcludeFilter

431e6aa

cleanup

20b5be4

vercel bot deployed to Preview February 5, 2020 01:42 View deployment

rename file to compare-runs.js

6f8df45

vercel bot deployed to Preview February 5, 2020 01:43 View deployment

paulirish commented Feb 5, 2020

View reviewed changes

paulirish changed the title ~~misc(collect-timings): report metrics, add gather/audit modes, etc.~~ misc(compare-runs): report metrics, add gather/audit modes, etc. Feb 5, 2020

paulirish changed the title ~~misc(compare-runs): report metrics, add gather/audit modes, etc.~~ misc(compare-runs): report metrics, add gather/audit modes, etc Feb 5, 2020

connorjclark reviewed Feb 5, 2020

View reviewed changes

connorjclark approved these changes Feb 5, 2020

View reviewed changes

exclude key by default

05ab2e5

vercel bot deployed to Preview February 6, 2020 00:26 View deployment

paulirish added the land-when-ci-is-green label Feb 6, 2020

connorjclark removed the needs-priority label Feb 6, 2020

paulirish added 2 commits February 6, 2020 16:59

eslint

cc7b137

drop mkdirp

96da782

vercel bot deployed to Preview February 7, 2020 01:12 View deployment

lint

385a668

vercel bot deployed to Preview February 28, 2020 18:48 View deployment

devtools-bot merged commit c59c0bb into master Feb 28, 2020

devtools-bot deleted the timingshax branch February 28, 2020 19:21

devtools-bot removed the land-when-ci-is-green label Feb 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

misc(compare-runs): report metrics, add gather/audit modes, etc #10296

misc(compare-runs): report metrics, add gather/audit modes, etc #10296

paulirish commented Feb 4, 2020 •

edited by connorjclark

Loading

connorjclark Feb 4, 2020

connorjclark Feb 4, 2020

paulirish Feb 5, 2020

connorjclark Feb 4, 2020 •

edited

Loading

paulirish Feb 5, 2020

connorjclark Feb 4, 2020

paulirish Feb 5, 2020

connorjclark Feb 5, 2020

connorjclark Feb 4, 2020

paulirish Feb 5, 2020

connorjclark Feb 4, 2020

connorjclark Feb 4, 2020

connorjclark Feb 4, 2020

paulirish Feb 5, 2020

paulirish Feb 5, 2020

connorjclark Feb 4, 2020

paulirish Feb 5, 2020

connorjclark Feb 4, 2020

paulirish Feb 5, 2020

connorjclark commented Feb 4, 2020 •

edited

Loading

paulirish left a comment

paulirish Feb 5, 2020

paulirish Feb 5, 2020

paulirish Feb 5, 2020

paulirish Feb 5, 2020

paulirish Feb 5, 2020

paulirish Feb 5, 2020

paulirish Feb 5, 2020

paulirish Feb 5, 2020

connorjclark Feb 5, 2020

misc(compare-runs): report metrics, add gather/audit modes, etc #10296

misc(compare-runs): report metrics, add gather/audit modes, etc #10296

Conversation

paulirish commented Feb 4, 2020 • edited by connorjclark Loading

--gather and --audit modes, and visual progress bar

metrics are reported

Coefficient of variation is reported

other

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

connorjclark Feb 4, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

connorjclark commented Feb 4, 2020 • edited Loading

paulirish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulirish commented Feb 4, 2020 •

edited by connorjclark

Loading

`--gather` and `--audit` modes, and visual progress bar

connorjclark Feb 4, 2020 •

edited

Loading

connorjclark commented Feb 4, 2020 •

edited

Loading