ESLint failing tests

[Cellule]

Hi, recently I have encountered quite a few problems with ESLint because I used babel-eslint. I would really like to keep using it for its better support for es6 and jsx.
I wanted to have an overview of the problems with babel-eslint so I executed ESLint test suite using babel-eslint and I wanted to share my results here.

The test suite found 115 failing test cases affecting 26 rules.
the rules are

• block-scoped-var
• brace-style
• comma-spacing
• complexity
• consistent-return
• default-case
• indent
• key-spacing
• no-catch-shadow
• no-div-regex
• no-empty-class
• no-ex-assign
• no-extra-parens
• no-fallthrough
• no-inline-comments
• no-multi-spaces
• no-regex-spaces
• no-shadow-restricted-names
• no-undefined
• padded-blocks
• space-before-blocks
• space-infix-ops
• spaced-line-comment
• valid-jsdoc
• vars-on-top
• wrap-regex

and you can see the detailed errors in the gist below (I enhanced a bit eslint-tester to have more information about failing tests)
https://gist.github.com/Cellule/37e1231437b1a28cdf46

I hope this can help solve most discrepancies between espree and babel-eslint.
If you want I can share how I ran the test suite since there is no official support at the moment.

[sebmck]

If you want I can share how I ran the test suite since there is no official support at the moment.

Sure, this would be great. Bit hard to ensure that these issues actually get fixed unless I can repeatedly run the test suite.

[Cellule]

Right now I am working with eslint-tester to provide a way to change the config for all test cases https://github.com/eslint/eslint-tester/issues/19

However, in order to run the tests, I simply added a field on the ESlintTester prototype that would be used to set the parser for the tests.
Then I installed babel-eslint in the ESlint repo and I set the new ESlintTester field in tests\lib\rules.js to babel-eslint like

var ESLintTester = require("eslint-tester");
ESLintTester.defaultParser = "babel-eslint";

Then every test ran after that point would use babel-eslint instead of espree

[gabrielecirulli]

Please don't quote me on this, but might it be a change in the eslint api? I noticed this started to happen when eslint went from 0.16 to 0.17. We just locked the version to 0.16 and everything was fine, but then babel-eslint got update and the error got triggered again. It also happens on eslint 0.19.

[gabrielecirulli]

In my case the main problem seems to be recognizing that JSX is using certain variables. It will trigger no-unused-var for all JSX elements.

[gabrielecirulli]

Locking to the following versions: [email protected] [email protected] seems to temporarily remove the issue.

[jsg2021]

+1 for this

[hzoo]

@Cellule I'm probably just bad at following your instructions (and haven't been able to get this working myself) - have you been able to get farther on this? It definitely would be awesome to be able to run against all ESLint tests.

[Cellule]

The problem is that my change to have a default configuration in eslint-tester hasn't been approved yet. see eslint/eslint-tester#20

So what you can do is clone eslint then run npm i cellule/eslint-tester#default_config babel-eslint to have my version.
Then go to tests\lib\rules\block-scoped-var.js (the first rule) and add at the top of the file

var ESLintTester = require("eslint-tester");
ESLintTester.setDefaultConfig({
    parser: "babel-eslint"
});

and then run npm test.

It's not very clean, but it gets the job done.
As of today I found 77 tests cases failing

[hzoo]

Ok I'l try it out tomorrow - and looks like there's less tests failing!

[Cellule]

If you want you can try my branch
https://github.com/Cellule/babel-eslint/tree/eslint_tests
to use it, you need to have a copy of eslint and run npm link in eslint directory. Then run npm link eslint in babel-eslint directory. That way you would have access to the test files.
After that, npm test in babel-eslint will run the test suite of eslint.

It's not perfect, I've done this real quick tonight and there are some false negative (some tests also fails with espree), but at least you can see some tests with minimal setup.

[ilyavolodin]

Just to let you guys know, I just published new version of eslint-tester that includes ability to provide default configuration, including parser that @Cellule submitted.

[hzoo]

Ok thanks @ilyavolodin (and @Cellule), I'l check it out. So I need to be able to run somewhere

var ESLintTester = require("eslint-tester");
ESLintTester.setDefaultConfig({
    parser: "babel-eslint"
});

and then run all the ESLint tests? What do you guys think would be the best way to integrate the tests into the project so we can test against regressions as well as fix the already failing tests? - ESLint is already a devDependency

[Cellule]

The problem is that having eslint as a dependency doesn't come with the tests. So right now no matter how you look at it you need to have your own copy of eslint and then run the tests. You can either modifiy eslint to run the tests with babel-eslint or use my (very hacky) branch of babel-eslint to run the eslint test suite (which is a bit less reliable).

[Cellule]

Since no matter how you look at it, you will need a copy of eslint. I figured it would be simpler to fork eslint and run the tests.
So I made a branch for eslint with a fairly simple way to run eslint test suite with babel-eslint
you can see the diff here

[hzoo]

Looks simple enough - my idea was to use a git sub module or something

// 88 failing tests

[hzoo]

@Cellule not really sure what we can do - a lot of the failing tests aren't an issue - strict mode, etc.

[Cellule]

We're going to have to find a way to mark or disregard these tests we know are false negatives. Right now I don't have any idea since we don't want to edit the source code of the tests.

[kaicataldo]

Looking into this and this is the current output from the tests:

> [email protected] eslint /Users/kaicataldo/Code/babel-eslint
> cd eslint && mocha -c tests/lib/rules/*.js -r ../eslint-tester.js

Use babel-eslint for test suite

assert.js:90
  throw new assert.AssertionError({
  ^
AssertionError: Should have no errors but had 1: [ { fatal: true,
    severity: 2,
    message: 'Parsing error: \'with\' in strict mode',
    line: 1,
    column: 2 } ]
    at testValidTemplate (/Users/kaicataldo/Code/babel-eslint/eslint/lib/testers/rule-tester.js:324:20)
    at Function.<anonymous> (/Users/kaicataldo/Code/babel-eslint/eslint/lib/testers/rule-tester.js:405:21)
    at Function.RuleTester.it (/Users/kaicataldo/Code/babel-eslint/eslint/lib/testers/rule-tester.js:193:19)
    at /Users/kaicataldo/Code/babel-eslint/eslint/lib/testers/rule-tester.js:404:28
    at Array.forEach (native)
    at Function.<anonymous> (/Users/kaicataldo/Code/babel-eslint/eslint/lib/testers/rule-tester.js:403:24)
    at Function.RuleTester.describe (/Users/kaicataldo/Code/babel-eslint/eslint/lib/testers/rule-tester.js:189:19)
    at Object.RuleTester.run (/Users/kaicataldo/Code/babel-eslint/eslint/lib/testers/rule-tester.js:402:20)

Been trying to figure out where that comes from...Will keep digging

[hzoo]

It's prob b/c babel-eslint has strict mode by default (since you are using modules, etc) and eslint uses with in it's tests

[kaicataldo]

@hzoo Ah, that makes sense...Hm, so how did they ever work?

[hzoo]

It didn't actually - there were always failing tests

[kaicataldo]

It sounded like it was running some of the tests, though - or maybe I misunderstood. What I pasted above is the entire output at the moment - doesn't run any of them :(

[hzoo]

Yeah I guess it changed in the time since I ran them? It looks like it's just stopping on the first failing one rather than running it all

[kaicataldo]

I see - yeah, these error messages are not super helpful, unfortunately. When I set strict to false it reports a different parsing error with no real error message.

[hzoo]

Going to close this in favor of #739