test: general improvements to vm tests #14458

jasnell · 2017-07-24T19:29:51Z

General improvements to vm tests

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

test

addaleax · 2017-07-24T19:32:07Z

test/parallel/test-vm-context.js

 context = vm.createContext({ 'foo': 'bar', 'thing': 'lala' });
 assert.strictEqual('bar', context.foo);
 assert.strictEqual('lala', context.thing);

-console.error('test updating context');
+// Rest updating context


Rest is a typo, right?

yep, should have been Test

Trott · 2017-07-24T20:43:30Z

Not objecting enough to stop it, but I'm not a fan of removing console.* calls in tests as a supposed improvement. Those are often super helpful when tests start failing in CI in ways that are difficult to reproduce locally. Seems to favor abstract ideas about how tests are "supposed" to be written over practical considerations.

jasnell · 2017-07-24T21:27:38Z

I've been considering introducing a new common.log() that would be a non-op unless a NODE_TEST_LOG=1 environment variable is set... that would accomplish the same basic thing without introducing the unnecessary noise by default.

Trott · 2017-07-24T22:58:12Z

I've been considering introducing a new common.log() that would be a non-op unless a NODE_TEST_LOG=1 environment variable is set... that would accomplish the same basic thing without introducing the unnecessary noise by default.

What is the downside of console.log() statements that we are trying to avoid? What is the value we get in return for the added complexity?

(Feel free to take this offline if it's a distraction from the PR generally.)

jasnell · 2017-07-25T05:13:05Z

most of the time the console.log output is generally quite useless in the tests that exist currently. For instance, printing console.log('ok') unconditionally when the test exits, or calling console.log() within unnecessary event handlers, with content that is not very useful, even for simple debugging.

Trott · 2017-07-25T06:05:02Z

most of the time the console.log output is generally quite useless in the tests that exist currently. For instance, printing console.log('ok') unconditionally when the test exits, or calling console.log() within unnecessary event handlers, with content that is not very useful, even for simple debugging.

If most of the logging is useless (which is an assessment I do not agree with), how does creating a whole new way to control logging improve anything? It seems that it would be better to improve or remove existing log messages.

Trott · 2017-07-25T06:06:48Z

Although looking more at these specific logging messages...yeah, not sure I get the utility of them. Sorry I said anything here. I should have saved it for something else where the log messages might have value. 😆

Trott

LGTM if CI is green.

jasnell · 2017-07-25T14:51:31Z

@Trott ... I definitely understand where you're coming from. Introducing a new common.log() type of mechanism and converting over gives us an opportunity to do two things: 1) improve the existing output messages to make them more useful and 2) improve the tests overall. As is evidenced by the recent round of test improvements I've been making, there are a range of improvements that can be made.

lpinca · 2017-07-26T06:39:28Z

test/parallel/test-vm-syntax-error-message.js

-p.stderr.on('data', function(data) {
-  assert.fail(`Unexpected stderr data: ${data}`);
-});
+p.stderr.on('data', common.mustNotCall);


The function should be invoked: common.mustNotCall().

lpinca · 2017-07-26T06:40:09Z

test/parallel/test-vm-syntax-error-stderr.js

-p.stdout.on('data', function(data) {
-  assert.fail(`Unexpected stdout data: ${data}`);
-});
+p.stdout.on('data', common.mustNotCall);


Trott

Needs nits from @lpinca before landing.

jasnell · 2017-07-31T19:03:11Z

Updated, rebased

updated since review, dismissing request for changes

jasnell · 2017-08-02T00:29:48Z

Rebased. CI before landing: https://ci.nodejs.org/job/node-test-pull-request/9445/

jasnell · 2017-08-02T00:36:14Z

Canceled the CI run... there appear to be some issues in CI-land (ping @nodejs/build)

jasnell · 2017-08-02T15:58:19Z

Trying new CI run again: https://ci.nodejs.org/job/node-test-pull-request/9454/

PR-URL: #14458 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Franziska Hinkelmann <[email protected]>

jasnell · 2017-08-03T18:02:36Z

Landed 4b23b42

PR-URL: #14458 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Franziska Hinkelmann <[email protected]>

nodejs-github-bot added the test Issues and PRs related to the tests. label Jul 24, 2017

addaleax approved these changes Jul 24, 2017

View reviewed changes

mscdex added the vm Issues and PRs related to the vm subsystem. label Jul 24, 2017

jasnell force-pushed the improve-vm-tests branch from 8295cbe to cc66afb Compare July 24, 2017 22:30

Trott approved these changes Jul 25, 2017

View reviewed changes

targos approved these changes Jul 25, 2017

View reviewed changes

cjihrig approved these changes Jul 25, 2017

View reviewed changes

lpinca reviewed Jul 26, 2017

View reviewed changes

Trott previously requested changes Jul 26, 2017

View reviewed changes

Trott mentioned this pull request Jul 29, 2017

test: refactor test-vm-new-script-new-context #14536

Closed

2 tasks

jasnell force-pushed the improve-vm-tests branch from cc66afb to 120bffe Compare July 31, 2017 19:02

tniessen approved these changes Aug 1, 2017

View reviewed changes

jasnell added 9 commits August 1, 2017 16:55

test: improve test-vm-syntax-error-stderr

c884f24

test: improve test-vm-run-in-new-context

ae2d714

test: improve test-vm-run-in-new-context

8e1ef69

test: improve test-vm-new-script-this-context

fea0067

test: improve test-vm-new-script-new-context

29e2576

test: improve test-vm-function-declaration

b8f2d88

test: test-vm-create-and-run-in-context

684c01f

test: improve test-vm-context

1556d3b

test: improve test-vm-context-async-script

716a19c

[Squash] Address nits

3fc4543

jasnell force-pushed the improve-vm-tests branch from 120bffe to 3fc4543 Compare August 2, 2017 00:28

fhinkel approved these changes Aug 3, 2017

View reviewed changes

jasnell closed this Aug 3, 2017

MylesBorins mentioned this pull request Sep 10, 2017

v8.5.0 proposal #15308

Merged

MylesBorins added the land-on-v6.x label Sep 20, 2017

MylesBorins mentioned this pull request Sep 20, 2017

v6.11.4 proposal #15506

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: general improvements to vm tests #14458

test: general improvements to vm tests #14458

jasnell commented Jul 24, 2017

addaleax Jul 24, 2017

jasnell Jul 24, 2017

Trott commented Jul 24, 2017

jasnell commented Jul 24, 2017

Trott commented Jul 24, 2017

jasnell commented Jul 25, 2017

Trott commented Jul 25, 2017

Trott commented Jul 25, 2017 •

edited

Loading

Trott left a comment

jasnell commented Jul 25, 2017

lpinca Jul 26, 2017

lpinca Jul 26, 2017

Trott left a comment

jasnell commented Jul 31, 2017

jasnell commented Aug 2, 2017

jasnell commented Aug 2, 2017

jasnell commented Aug 2, 2017

jasnell commented Aug 3, 2017

test: general improvements to vm tests #14458

test: general improvements to vm tests #14458

Conversation

jasnell commented Jul 24, 2017

Checklist

Affected core subsystem(s)

addaleax Jul 24, 2017

Choose a reason for hiding this comment

jasnell Jul 24, 2017

Choose a reason for hiding this comment

Trott commented Jul 24, 2017

jasnell commented Jul 24, 2017

Trott commented Jul 24, 2017

jasnell commented Jul 25, 2017

Trott commented Jul 25, 2017

Trott commented Jul 25, 2017 • edited Loading

Trott left a comment

Choose a reason for hiding this comment

jasnell commented Jul 25, 2017

lpinca Jul 26, 2017

Choose a reason for hiding this comment

lpinca Jul 26, 2017

Choose a reason for hiding this comment

Trott left a comment

Choose a reason for hiding this comment

jasnell commented Jul 31, 2017

jasnell commented Aug 2, 2017

jasnell commented Aug 2, 2017

jasnell commented Aug 2, 2017

jasnell commented Aug 3, 2017

Trott commented Jul 25, 2017 •

edited

Loading