tools, benchmark: test util benchmark #16050

sarahmeyer · 2017-10-06T20:11:48Z

coverage for benchmarking util

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

doc, benchmark

joyeecheung · 2017-10-06T20:49:07Z

benchmark/util/type-check.js

@@ -38,7 +38,7 @@ function main(conf) {
  const util = process.binding('util');
  const types = require('internal/util/types');

-  const n = (+conf.millions * 1e6) | 0;
+  const n = (+conf.n * 1e6) | 0;


This would increase n from 5 * 1e6 to 1e7 * 1e6, which is going to cost more time if the benchmark is run normally. Can you change the default n to 5e6 and use that value as-is(without multiplying) instead?

Ah yes, I agree that is a better solution. (Please keep the number coercion bits though.)

joyeecheung · 2017-10-06T20:50:25Z

test/parallel/test-benchmark-util.js

+
+const runBenchmark = require('../common/benchmark');
+
+runBenchmark('util', ['n=1', 'arguments=0']);


Setting arguments=0 would turn the arg in the test undefined. I believe we are good with n=1 only.

sarahmeyer · 2017-10-11T21:39:11Z

@joyeecheung thank you so much for your thoughtful feedback! ❤️

Fishrock123

LGTM, CI: https://ci.nodejs.org/job/node-test-pull-request/10632/

(which probably won't fully pass regardless but hey)

Fishrock123 · 2017-10-11T22:36:14Z

Hmm, timed out on SmartOS. [1]

not ok 296 parallel/test-benchmark-util
  ---
  duration_ms: 120.492
  severity: fail
  stack: |-
    timeout
  ...

Edit: looks like we are taking too long still for some reason, the longest other benchmark test is test-benchmark-path:

ok 198 parallel/test-benchmark-path
  ---
  duration_ms: 59.499
  ...

Fishrock123 · 2017-10-11T23:27:45Z

My guess is that it just spawns too many child processes for slow machines. It still takes 10s on my i7 2.6ghz.

So, this just got a bit more complex, but to get this to reliably pass on all systems there are two options (I think):

Add test-benchmark-util: PASS,FLAKY to [$system==solaris] in parallel.status. (Simple)
Modify benchmark/_cli.jsto be able to take a more specific "category" format, such as util/inspect and then separate this test into multiple files. (More work)

Maybe @nodejs/testing can advise on whether option 1 is reasonable.

Trott

Thanks for taking this on! Before this lands, we need some more options added to that array that is the second argument to runBenchmark(). Basically, anything that has more than one option in any of the util benchmark files will need an option passed so that we aren't running every combination of options for each file. While that would be a thorough test, it will time out in CI for some hosts. The goal of these benchmark tests are to flag benchmark changes that totally break everything, rather than thoroughly test the files, so it's OK if there is code in the benchmarks that end up being skipped because of this.

So, you'll probably want to make the array ['argument=false', 'input=', 'method=Array', 'n=1', 'option=none'] plus a few more options in there.

To get an idea of how the options affect the way the benchmark runs, you can try adding elements to the array one at a time and see how it affects the output of ./node test/parallel/test-benchmark-util.js.

There may be other pitfalls that pop up--we'll see. Be persistent. These benchmark tests aren't difficult to write, but they do require a bit of fussing and patience to get right.

Trott · 2017-10-12T00:17:29Z

Maybe @nodejs/testing can advise on whether option 1 is reasonable.

@Fishrock123 I'd be a pretty hard "no" on option 1. Option 2 could work, but the usual way we handle this is to set more items in the settings array so that the benchmark has fewer combinations of settings to run. See my comment above and/or look at existing test-benchmark-* files for examples. I'd encourage @sarahmeyer to try that first. I'm pretty certain that will work and will probably be easier than digging into _cli.js.

(This was a "good second commit" task from Code + Learn. It's totally do-able but likely to require more back-and-forth and experimentation than the first tasks given at the event.)

joyeecheung · 2017-10-12T02:58:23Z

@Trott @Fishrock123 If we are talking about running one file at a time, run.js --filter inspect util would do the job already, we only need to modify runBenchmark so it takes --filter arguments, which should be pretty easy.

Also this reminds me of the child_process.spawn in other tests that have caused hanging problems before. I am not terribly familiar with tools/test.py but I think it should be possible for the javascript test script to talk to it's parent so we can coordinate the scheduling of these sub-child processes? I will try to take a look later.

joyeecheung · 2017-10-12T03:24:40Z

@Trott @Fishrock123 OK I think this is not related to "too many processes at the same time" because run.js would lauch the benchmark one by one. I think we can still try to run all the combinations on faster machines while only run a sample of them on slower ones.

(We don't have a common.isSmartOS right now....should we implement one?)

Trott · 2017-10-12T06:14:50Z

I think we can still try to run all the combinations on faster machines while only run a sample of them on slower ones.

With the right options, this test will run plenty fast. Just given the options I put in my comment above (which is a partial list--still more to add, but I left that as an exercise for the contributor), the run time on my laptop for the test is reduced from 19 seconds to less than 3 seconds. With the right arguments this test will be no problem at all.

(We don't have a common.isSmartOS right now....should we implement one?)

If we wanted to do that (which we don't in this case!), we could use common.isSunOS. SmartOS started life as Solaris, I believe.

@Trott

Prefer @Trott's suggestions

Trott · 2017-10-28T01:22:35Z

I went ahead and made the changes involving the arguments/settings. Hope that's OK. This should be good to land, I think. @joyeecheung @Fishrock123 PTAL

CI: https://ci.nodejs.org/job/node-test-pull-request/11032/

Trott · 2017-10-28T01:22:59Z

(Oh, run time is now down to less than 1.5 seconds locally for me. 🎉)

settings are now implemented here

Create a minimal test for the util benchmark files. PR-URL: nodejs#16050 Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anatoli Papirovski <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

Trott · 2017-10-28T20:57:38Z

Landed in 50fe1a8.

Thanks for the contribution! 🎉

Create a minimal test for the util benchmark files. PR-URL: #16050 Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anatoli Papirovski <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

gibfahn · 2017-10-30T21:42:24Z

Should this be backported to v8.x-staging? If yes please follow the guide and raise a backport PR, if no let me know or add the dont-land-on label.

cc/ @joyeecheung @Trott regarding suitability for backporting.

The commit cherry-picks fine, but we get this test failure:

=== release test-benchmark-util ===                                            
Path: parallel/test-benchmark-util
util/format.js
util/format.js type="" n=1: 2,428.8410299257503

util/inspect-array.js
util/inspect-array.js type="" len=100000 n=1: 54.0944640807349

util/inspect-proxy.js
util/inspect-proxy.js n=1: 456.76616860883644

util/inspect.js
util/inspect.js option="none" method="Array" n=1: 316.97499352578575

util/normalize-encoding.js
util/normalize-encoding.js n=1 input="": 6,362.861251447551

util/type-check.js
/home/gib/node/benchmark/util/type-check.js:40
  const arg = args[conf.type][conf.argument];
                             ^

TypeError: Cannot read property 'false' of undefined
    at main (/home/gib/node/benchmark/util/type-check.js:40:30)
    at Benchmark.process.nextTick (/home/gib/node/benchmark/common.js:34:28)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)
    at Function.Module.runMain (module.js:676:11)
    at startup (bootstrap_node.js:187:16)
    at bootstrap_node.js:608:3
assert.js:42
  throw new errors.AssertionError({
  ^

AssertionError [ERR_ASSERTION]: 1 === 0
    at ChildProcess.child.on (/home/gib/node/test/common/benchmark.js:25:12)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
Command: out/Release/node /home/gib/node/test/parallel/test-benchmark-util.js

Trott · 2017-10-30T23:39:48Z

I wouldn't bother backporting this unless someone wanted to do it as an exercise. It's great for code changes going forward, but we don't need it on old release lines. We've gotten by just fine thus far without it. :-D

Create a minimal test for the util benchmark files. PR-URL: nodejs/node#16050 Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anatoli Papirovski <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]>

nodejs-github-bot added benchmark Issues and PRs related to the benchmark subsystem. test Issues and PRs related to the tests. util Issues and PRs related to the built-in util module. labels Oct 6, 2017

joyeecheung requested changes Oct 6, 2017

View reviewed changes

Trott added the code-and-learn Issues related to the Code-and-Learn events and PRs submitted during the events. label Oct 6, 2017

Trott assigned Trott and unassigned Trott Oct 10, 2017

Fishrock123 previously approved these changes Oct 11, 2017

View reviewed changes

Trott previously requested changes Oct 12, 2017

View reviewed changes

sarahmeyer and others added 3 commits October 27, 2017 16:35

tools, benchmark: test util benchmark

88c786d

tools, benchmark: solidify benchmarking arguments

7befb1c

squash: arguments

78ebc37

Trott force-pushed the test_util_benchmark branch from c20929b to 78ebc37 Compare October 28, 2017 01:17

apapirovski approved these changes Oct 28, 2017

View reviewed changes

joyeecheung approved these changes Oct 28, 2017

View reviewed changes

gireeshpunathil approved these changes Oct 28, 2017

View reviewed changes

Trott closed this Oct 28, 2017

jasnell mentioned this pull request Oct 30, 2017

Proposal: 2017-10-31, Version 9.0.0 (Current) #15135

Merged

gibfahn added the backport-requested-v8.x label Oct 30, 2017

Trott added dont-land-on-v4.x and removed backport-requested-v8.x labels Oct 30, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools, benchmark: test util benchmark #16050

tools, benchmark: test util benchmark #16050

sarahmeyer commented Oct 6, 2017

joyeecheung Oct 6, 2017

Fishrock123 Oct 9, 2017 •

edited

Loading

joyeecheung Oct 6, 2017

sarahmeyer commented Oct 11, 2017

Fishrock123 left a comment

Fishrock123 commented Oct 11, 2017 •

edited

Loading

Fishrock123 commented Oct 11, 2017 •

edited

Loading

Trott left a comment

Trott commented Oct 12, 2017

joyeecheung commented Oct 12, 2017

joyeecheung commented Oct 12, 2017

Trott commented Oct 12, 2017 •

edited

Loading

Trott commented Oct 28, 2017

Trott commented Oct 28, 2017

Trott commented Oct 28, 2017

gibfahn commented Oct 30, 2017

Trott commented Oct 30, 2017


		const runBenchmark = require('../common/benchmark');

		runBenchmark('util', ['n=1', 'arguments=0']);

tools, benchmark: test util benchmark #16050

tools, benchmark: test util benchmark #16050

Conversation

sarahmeyer commented Oct 6, 2017

Checklist

Affected core subsystem(s)

joyeecheung Oct 6, 2017

Choose a reason for hiding this comment

Fishrock123 Oct 9, 2017 • edited Loading

Choose a reason for hiding this comment

joyeecheung Oct 6, 2017

Choose a reason for hiding this comment

sarahmeyer commented Oct 11, 2017

Fishrock123 left a comment

Choose a reason for hiding this comment

Fishrock123 commented Oct 11, 2017 • edited Loading

Fishrock123 commented Oct 11, 2017 • edited Loading

Trott left a comment

Choose a reason for hiding this comment

Trott commented Oct 12, 2017

joyeecheung commented Oct 12, 2017

joyeecheung commented Oct 12, 2017

Trott commented Oct 12, 2017 • edited Loading

Trott commented Oct 28, 2017

Trott commented Oct 28, 2017

Trott commented Oct 28, 2017

gibfahn commented Oct 30, 2017

Trott commented Oct 30, 2017

Fishrock123 Oct 9, 2017 •

edited

Loading

Fishrock123 commented Oct 11, 2017 •

edited

Loading

Fishrock123 commented Oct 11, 2017 •

edited

Loading

Trott commented Oct 12, 2017 •

edited

Loading