add CLI option to syntax check script #2411

bahamas10 · 2015-08-17T21:35:58Z

original PR: nodejs/node-v0.x-archive#9447
original discussion: nodejs/node-v0.x-archive#9426

examples

$ cat ./good.js
var foo = 'bar';
$ ./out/Release/iojs -c ./good.js
$ echo $?
0

$ cat ./bad.js
var foo bar;
$ ./out/Release/iojs -c ./bad.js
var foo bar;
        ^^^

SyntaxError: Unexpected identifier
    at startup (node.js:99:13)
    at node.js:961:3
$ echo $?
1

tests pass

$ ./out/Release/iojs test/parallel/test-cli-syntax.js 
calling /Users/dave.eddy/dev/node-1/out/Release/iojs -c /Users/dave.eddy/dev/node-1/test/fixtures/syntax/good_syntax.js
ok
calling /Users/dave.eddy/dev/node-1/out/Release/iojs --check /Users/dave.eddy/dev/node-1/test/fixtures/syntax/good_syntax.js
ok
calling /Users/dave.eddy/dev/node-1/out/Release/iojs -c /Users/dave.eddy/dev/node-1/test/fixtures/syntax/good_syntax_shebang.js
ok
calling /Users/dave.eddy/dev/node-1/out/Release/iojs --check /Users/dave.eddy/dev/node-1/test/fixtures/syntax/good_syntax_shebang.js
ok
calling /Users/dave.eddy/dev/node-1/out/Release/iojs -c /Users/dave.eddy/dev/node-1/test/fixtures/syntax/bad_syntax.js
ok
calling /Users/dave.eddy/dev/node-1/out/Release/iojs --check /Users/dave.eddy/dev/node-1/test/fixtures/syntax/bad_syntax.js
ok
calling /Users/dave.eddy/dev/node-1/out/Release/iojs -c /Users/dave.eddy/dev/node-1/test/fixtures/syntax/bad_syntax_shebang.js
ok
calling /Users/dave.eddy/dev/node-1/out/Release/iojs --check /Users/dave.eddy/dev/node-1/test/fixtures/syntax/bad_syntax_shebang.js
ok

bnoordhuis · 2015-08-18T00:28:28Z

src/node.js

+        if (process._syntax_check_only != null) {
+            var vm = NativeModule.require('vm');
+            var fs = NativeModule.require('fs');
+            var source = fs.readFileSync(process.argv[1], 'utf-8').replace(/^#.*/, '');


I see a couple of issues here:

It won't work for filenames without an extension (node app vs. node app.js)

It won't skip the UTF-8 BOM (byte order marker) like require() does.

The regular expression doesn't match the one in lib/module.js that matches the string #!.

It's probably unavoidable that fixing 2 and 3 results in some duplicated code but it should come with regression tests that ensure the implementations won't diverge.

bnoordhuis · 2015-08-18T00:31:54Z

Left some comments. Basic premise of this PR seems alright to me.

bahamas10 · 2015-08-18T04:46:51Z

Thanks @bnoordhuis for the review.

I've updated the code to use Module._resolveFilename on process.argv[1] to fix the node app vs node app.js problem.
I've exposed stripBOM as Module._stripBOM which is used by this code in node.js (to avoid repeated code)
updated regex to match that in lib/module.js
updated tests to use const and 'use strict';... it turns out I was testing joyent/node and not nodejs/node accidentally before... so i've made sure this all passes make test in this repo
removed all logging on success in the test supplied

Additionally I've added to test for file vs file.js, as well as files that don't exist.

thefourtheye · 2015-08-18T10:15:30Z

src/node.cc

@@ -2843,6 +2844,11 @@ void SetupProcessObject(Environment* env,
    READONLY_PROPERTY(process, "_print_eval", True(env->isolate()));
  }

+  // -c, --check
+  if (syntax_check_only) {
+    READONLY_PROPERTY(process, "_syntax_check_only", True(env->isolate()));


I would prefer using consistent camel casing. I am not sure if we can fix the _print_eval above. cc @bnoordhuis

I'd be happy to change this if everyone agrees this should be done - I just tried to match the style I saw in this file.

It's consistent with _print_eval. I'm fine with it.

Oh okay then. I was just worried that we would introduce non camel case identifiers in to the js land.

Is this needed? the syntax check won't actually launch the script so there will never be access to process._syntax_check_only.

wait. nm. was thinking from the users' perspective. forgot about internal tooling.

rvagg · 2015-08-18T12:13:31Z

Hey @bahamas10, welcome!

This does seem like something that could easily be achieved by a userland module, surely there's already something in npm that does this, so what's the case for putting it in core?

bahamas10 · 2015-08-18T16:16:15Z

Hey @rvagg

The indirect answer is here nodejs/node-v0.x-archive#9426

But to be more specific, to syntax check scripts:

bash -n somescript
ruby -c somescript.rb
perl -c somescript.pl

etc.

Most interpreted language interpreters support some way of saying "are the contents of the file given valid for interpretation?". This does not check style nor does it lint the content - just syntax.

However, I realize the argument that "a bunch of people are doing it" is not sufficient.

The node core is already the source of truth when it comes to compiling JavaScript source and determining if it is valid. For example, an npm module would be nothing more than a thin wrapper around the vm module and new vm.Script(...). Since the node core is already doing the heavy lifting, and it itself contains the logic to determine the validity of JavaScript source code, it alone should be responsible for it.

As an example, imagine the following script (downloaded from npm or wherever) called syntax-check

#!/usr/bin/env node
var vm = require('vm');
var fs = require('fs');
var source = fs.readFileSync(process.argv[2], 'utf8').replace(/^#.*/, '');
new vm.Script(source, {displayErrors: true});

You could then call this script in the following way to achieve almost the same results as this PR

$ ./syntax-check syntax-check
$ echo $?
0

However, this only checks the validity of it against whichever node binary is available first in the users $PATH (because of the shebang). So test other versions of node a user must run something like

$ ~/dev/node/out/Release/node "$(npm config get prefix)/bin/syntax-check" script.js

This requires a lot of knowledge about the underlying system, and also forces a specific interpreter for the script, going against what is in the shebang (ie. what the user intended it to be executed with) - thus running a script downloaded from npm for a specific version of node with a different version of node.

I think it makes the most sense to cut out anything in the middle - any 3rd party script - and just ask the node binary itself "can you run this?"

bahamas10 · 2015-08-18T16:40:36Z

moved stripBOM to lib/internal/module.js and use it in both lib/module.js and src/node.js
syntax_args => syntaxArgs
args.concat([file]) => args.concat(file)

make test passes

bnoordhuis · 2015-08-18T18:05:08Z

The commit log should conform to the guidelines from CONTRIBUTING.md but otherwise LGTM. Let's see what the CI thinks: https://jenkins-iojs.nodesource.com/job/node-test-pull-request/104/

bahamas10 · 2015-08-18T19:49:03Z

updated commit message based on CONTRIBUTING.md guidelines.

rvagg · 2015-08-18T23:40:28Z

OK, I'm -0 on this, I won't hold up merging if other collaborators feel strongly enough that this is a good thing to have

bahamas10 · 2015-08-26T17:10:25Z

(merge conflicts resolved and rebased with latest master)

bahamas10 · 2015-09-04T17:16:15Z

@rvagg I don't feel comfortable about this being merged if you are having reservations about it. Is there anything specific about this you don't like? or is it an overall negative feeling about adding something extra to the node core?

If you're available I'm happy to talk about this and make sure we are both on the same side with regards to this change.

bnoordhuis · 2015-09-04T20:47:07Z

I didn't comment on it last time but FWIW, it seems like a reasonable addition to me with precedent in other scripting environments. Putting it in a npm package makes it markedly less convenient.

rvagg · 2015-09-05T01:32:04Z

yeah, sorry for not responding earlier, this is fine by me, the fact that other runtimes have the same functionality makes this easier to swallow

rvagg · 2015-09-05T01:33:23Z

https://ci.nodejs.org/job/node-test-pull-request/254/ again, the last one looked pretty flaky

thefourtheye · 2015-09-05T02:19:17Z

src/node.js

+        // check if user passed `-c` or `--check` arguments to Node.
+        if (process._syntax_check_only != null) {
+          var vm = NativeModule.require('vm');
+          var fs = NativeModule.require('fs');


Use const wherever the values are assigned once and never changed again.

this could go either way cause it's not at all consistent in the file, we might be shifting that way but there's only one use of const amongst the 20 NativeModule.require calls already in this file, someone's probably going to do a round of cleanup at some point and this change could be done then

@rvagg Oh, okay. I thought if we allowed the new code being landed to follow this rule strictly, then we just have to cleanup only the old code.

rvagg · 2015-09-05T03:13:02Z

all green, https://ci.nodejs.org/job/node-test-commit/506/, that's pretty good given the recent state of our suite, lgtm

thefourtheye · 2015-09-16T06:13:18Z

test/parallel/test-cli-syntax.js

+
+const common = require('../common');
+
+var node = process.argv[0];


Use process.execPath

@bahamas10 could you change this please?

Fishrock123 · 2015-09-16T13:36:27Z

Seems fine.

yosuke-furukawa · 2015-09-16T18:15:24Z

+1

rvagg · 2015-09-25T06:06:07Z

@bahamas10 I see 2 outstanding items, pointed them out inline, then I'll merge this and we should be able to get it into v4.x

bahamas10 · 2015-09-26T01:49:04Z

thanks @thefourtheye for the problems you've spotted. @rvagg i've force pushed a new commit with the changes... here is the intermediate diff to easily spot the changes

$ gd
diff --git a/test/parallel/test-cli-syntax.js b/test/parallel/test-cli-syntax.js
index 059e3ef..20fdfdc 100644
--- a/test/parallel/test-cli-syntax.js
+++ b/test/parallel/test-cli-syntax.js
@@ -6,7 +6,7 @@ const path = require('path');

 const common = require('../common');

-var node = process.argv[0];
+var node = process.execPath;

 // test both sets of arguments that check syntax
 var syntaxArgs = [
@@ -49,9 +49,13 @@ var syntaxArgs = [
     var _args = args.concat(file);
     var c = spawnSync(node, _args, {encoding: 'utf8'});

-    // no output should be produced
+    // no stdout should be produced
     assert.equal(c.stdout, '', 'stdout produced');
-    assert(c.stderr, 'stderr not produced');
+
+    // stderr should have a syntax error message
+    var match = c.stderr.match(/^SyntaxError: Unexpected identifier$/m);
+    assert(match, 'stderr incorrect');
+
     assert.equal(c.status, 1, 'code == ' + c.status);
   });
 });
@@ -68,9 +72,13 @@ var syntaxArgs = [
     var _args = args.concat(file);
     var c = spawnSync(node, _args, {encoding: 'utf8'});

-    // no output should be produced
+    // no stdout should be produced
     assert.equal(c.stdout, '', 'stdout produced');
-    assert(c.stderr, 'stderr not produced');
+
+    // stderr should have a module not found error message
+    var match = c.stderr.match(/^Error: Cannot find module/m);
+    assert(match, 'stderr incorrect');
+
     assert.equal(c.status, 1, 'code == ' + c.status);
   });
 });

Fishrock123 · 2015-09-26T03:14:43Z

I'd say to please use const when you can if you add or update a variable. :)

rvagg · 2015-09-29T12:40:21Z

grf, bikeshed, can we defer the var vs const thing please @Fishrock123 & @thefourtheye? This 1/2 way state we're in is too messy and I haven't seen anything other than a vague idea amongst some collaborators that const is a must now.

PR-URL: #2411 Reviewed-By: Rod Vagg <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Evan Lucas <[email protected]>

rvagg · 2015-10-03T12:10:04Z

landed @ 2e6ece4 with commit summary modification

thanks @bahamas10 and congrats on first commit, and it's a significant one too!

bahamas10 · 2015-10-04T21:58:35Z

@rvagg + all. nice! thanks everyone.

jasnell · 2015-10-06T01:13:45Z

Nice to see this landed 👍

PR-URL: #2411 Reviewed-By: Rod Vagg <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Evan Lucas <[email protected]>

The first Node.js LTS release! See https://github.com/nodejs/LTS/ for details of the LTS process. * **icu**: Updated to version 56 with significant performance improvements (Steven R. Loomis) #3281 * **node**: - Added new `-c` (or `--check`) command line argument for checking script syntax without executing the code (Dave Eddy) #2411 - Added `process.versions.icu` to hold the current ICU library version (Evan Lucas) #3102 - Added `process.release.lts` to hold the current LTS codename when the binary is from an active LTS release line (Rod Vagg) #3212 * **npm**: Upgraded to npm 2.14.7 from 2.14.4, see release notes: https://github.com/npm/npm/releases/tag/v2.14.7 for full details (Kat Marchán) #3299 PR-URL: #3258

jays1204 · 2015-10-29T15:43:56Z

I understand that we are done talking about this topic. However, I am writing to ask you one question.
If test.js file has below contents then 'node test.js' works.

test.js

if (true) {
  return;
}

But 'node -c test.js' occurs error "SyntaxError: Illegal return statement".

'return' statement works in function. But node module system wraps files in a self-executing function. so "node test.js" works but "node -c test.js" not works.
Is this okay?
If I bother you then I'm sorry.

bnoordhuis · 2015-10-29T16:05:34Z

@jays1204 I'd consider that a bug. Can you open an issue?

jays1204 · 2015-10-29T16:46:50Z

@bnoordhuis already someone opened it :)

bahamas10 mentioned this pull request Aug 17, 2015

add CLI option to syntax check script, fixes #9426 nodejs/node-v0.x-archive#9447

Closed

bnoordhuis reviewed Aug 18, 2015
View reviewed changes

bahamas10 force-pushed the dave.eddy-1439847191 branch from 92f6f50 to 8ee8b0d Compare August 18, 2015 04:43

bahamas10 force-pushed the dave.eddy-1439847191 branch from 8ee8b0d to 3d10585 Compare August 18, 2015 07:31

thefourtheye reviewed Aug 18, 2015
View reviewed changes

bahamas10 force-pushed the dave.eddy-1439847191 branch from 3d10585 to d2809a3 Compare August 18, 2015 16:39

brendanashworth added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 18, 2015

bahamas10 force-pushed the dave.eddy-1439847191 branch from d2809a3 to 8c8998a Compare August 18, 2015 19:48

bahamas10 force-pushed the dave.eddy-1439847191 branch from 8c8998a to 64d8bf5 Compare August 26, 2015 17:09

thefourtheye reviewed Sep 5, 2015
View reviewed changes

thefourtheye reviewed Sep 16, 2015
View reviewed changes

src: add CLI option to syntax check script

03e1aa9

bahamas10 force-pushed the dave.eddy-1439847191 branch from 64d8bf5 to 03e1aa9 Compare September 26, 2015 01:46

rvagg mentioned this pull request Sep 30, 2015

enabling eslint's prefer-const rule #3118

Closed

rvagg pushed a commit that referenced this pull request Oct 3, 2015

node: add -c|--check CLI arg to syntax check script

2e6ece4

PR-URL: #2411 Reviewed-By: Rod Vagg <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Evan Lucas <[email protected]>

rvagg closed this Oct 3, 2015

jasnell mentioned this pull request Oct 8, 2015

Preparing v4.x for LTS #3252

Closed

29 tasks

jasnell pushed a commit that referenced this pull request Oct 8, 2015

node: add -c|--check CLI arg to syntax check script

9d65528

PR-URL: #2411 Reviewed-By: Rod Vagg <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Evan Lucas <[email protected]>

jasnell added the lts-landed-on-v4.x label Oct 8, 2015

jasnell mentioned this pull request Oct 8, 2015

Release proposal: v4.2.0 'Argon' (LTS) #3258

Closed

evanlucas mentioned this pull request Oct 29, 2015

src: wrap source before doing syntax check #3587

Merged


		const common = require('../common');

		var node = process.argv[0];

add CLI option to syntax check script #2411

add CLI option to syntax check script #2411

Conversation

bahamas10 commented Aug 17, 2015

Choose a reason for hiding this comment

bnoordhuis commented Aug 18, 2015

bahamas10 commented Aug 18, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rvagg commented Aug 18, 2015

bahamas10 commented Aug 18, 2015

bahamas10 commented Aug 18, 2015

bnoordhuis commented Aug 18, 2015

bahamas10 commented Aug 18, 2015

rvagg commented Aug 18, 2015

bahamas10 commented Aug 26, 2015

bahamas10 commented Sep 4, 2015

bnoordhuis commented Sep 4, 2015

rvagg commented Sep 5, 2015

rvagg commented Sep 5, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rvagg commented Sep 5, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fishrock123 commented Sep 16, 2015

yosuke-furukawa commented Sep 16, 2015

rvagg commented Sep 25, 2015

bahamas10 commented Sep 26, 2015

Fishrock123 commented Sep 26, 2015

rvagg commented Sep 29, 2015

rvagg commented Oct 3, 2015

bahamas10 commented Oct 4, 2015

jasnell commented Oct 6, 2015

jays1204 commented Oct 29, 2015

bnoordhuis commented Oct 29, 2015

jays1204 commented Oct 29, 2015