update babylon to babel/parser 7.x #30

firsttris · 2020-01-24T18:25:52Z

Hey jest-community,

i update babylon to babel/parser 7.x as suggested by @seanpoulter #29

now the typescriptt_parser.js is actually no longer relevant and could possibly be removed. also the typescript dependency in package.json.
since the babel_parser.js is now able to parse typescript as well.

~~i did not delete typescript_parser.js and corresponding test yet. (even if it would no longer be used after this PR)~~

i add a addtional test - babel_typescript_parser.test.js which executes babel_parser in typescript mode.

i checked that all tests are green, and fixed a few eslint issues.

if you are happy with this PR, i would proceed update the jest parser to new features like test.each.

regards
Tristan

stephtr

Why would we want to keep typescript_parser.js but not babylon_parser.js?

fixtures/parser_tests.js

src/parsers/babel_parser.js

connectdotz · 2020-01-26T17:29:48Z

@firsttris thanks for the PR, before diving into the code, here is a conversation over a year ago that you might find relevant.

IMHO, the most uncertain part of this migration is the plugin list. We used to load all the plugin (yeah, that is a bit crazy), now with an explicit plugin list, I am not sure if we will encounter issues that the user plugin list != our parser's. Will that be an issue for parsing their code? If it does, do we need to consider allow users to pass in their plugin list or even reading their babel config? (a can of worms...) Thoughts?

firsttris · 2020-01-26T19:51:31Z

reading the users babel config sounds complex to me . big fan of kiss principle.

if the parser supports more features that then user is using everything works well.
if the parser does not support features the user is using it will probably not parse the code and generate errors.

if adding all plugins has worked before.. why not continue this way?

i tried to add as many plugins as possible.
there was no measurable speed difference when adding all those plugin.
(did a tests but exacly the same ms when executing tests)

some plugins:

decorators
pipelineOperator
did not work

they outputed this errors:
The 'decorators' plugin requires a 'decoratorsBeforeExport' option, whose value must be a boolean. If you are migrating from Babylon/Babel 6 or want to use the old decorators proposal, you should use the 'decorators-legacy' plugin instead of 'decorators'.
'pipelineOperator' requires 'proposal' option whose value should be one of: 'minimal', 'smart', 'fsharp'

i actually added them as an array of strings?

also removed the typescript_parser as @stephtr suggested.

connectdotz · 2020-01-26T22:28:20Z

if adding all plugins has worked before.. why not continue this way?

reasonable starting point.

some plugins ... did not work

see discussion

i actually added them as an array of strings?

should be... but we should probably create some fixtures with some of these features, such as optional-chaining, dynamic import etc, to make sure they can be parsed.

stephtr · 2020-01-26T22:29:15Z

Will that be an issue for parsing their code? If it does, do we need to consider allow users to pass in their plugin list or even reading their Babel config?

There could be situations where Babel won't work with our configuration, but I'm quite sure that also the Babylon parser (as it is currently used) would have failed for them, it therefore wouldn't be a regression.

seanpoulter

Hurrah! I think this is a big step in the right direction! Thank you! There's only one question that @connectdotz and/or @stephtr didn't seem to already cover - is the CHANGELOG.md OK like that? I'll approve it now so we're not stuck with me being sloooooooooow. 😪

You've already had the discussion about use plugins. We'll probably want a seam / way for the user to provide options, but we can add options to the parse functions in another PR.

CHANGELOG.md

fixtures/parser_tests.js

src/parsers/babel_parser.js

src/__tests__/runner.test.js

seanpoulter · 2020-01-31T07:18:19Z

Oh!! I am tired and forgot to ask about Yarn. Does Yarn update package-lock.json now too or did you use npm? I'm not bothered by the politics of it, I just want to make sure we're all on board with our build process (which seems to be undocumented).

connectdotz

@stephtr and @seanpoulter already covered many grounds, I will just add a few relatively simple requests. I think we are quite close to merging this!

P.S. I assume you are going to do the actual parser change for the test.each in a separate PR, this is just to switch over to the babel-7 world, right?

src/parsers/babel_parser.js

src/parsers/index.js

firsttris · 2020-02-02T17:42:19Z

are going to do the actual parser change for the test.each in a separate PR, this is just to switch over to the babel-7 world, right?

yes, in those enterprises they told to always keep my PR's small :)

i don't wanted to extend two parsers.

firsttris · 2020-02-02T17:45:02Z

Oh!! I am tired and forgot to ask about Yarn. Does Yarn update package-lock.json now too or did you use npm? I'm not bothered by the politics of it, I just want to make sure we're all on board with our build process (which seems to be undocumented).

i think i used npm, as far as i know package-lock.json is only created by npm

connectdotz · 2020-02-03T15:49:16Z

i think i used npm, as far as i know package-lock.json is only created by npm

I wonder if we allow both yarn and npm lock files, will we encounter a situation that developers will end up with different versions of the dependencies based on their pm preference? Or otherwise we will ask developers to update both lock files by running boh yarn and npm? 🤔 Maybe it is best to standardize on the PM, which right now is yarn...

src/parsers/index.js

connectdotz

@firsttris lots of good change here 👍, a few notes:

plugin customization: nice feature, but suggest we still have a solid default, more detail inline
readme: great start, some minor comment inline, might need to be updated if plugin customization changed.
npm support: personally I prefer us to simply standardize on yarn. But if you would like to add npm support that would be fine as long as there is a mechanism to sync the 2 lock files automatically so they won't get out of sync.

connectdotz · 2020-02-08T17:35:34Z

src/parsers/babel_parser.js

+export const plugins = ['jsx', 'classProperties'];
+
+// Its not possible to use the parser with flow and typescript active at the same time
+export const parseJs = (file: string, data: ?string, additionalPlugins: string[] = []): ParseResult =>
+  parse(file, data, {plugins: [...plugins, ...additionalPlugins, 'flow']});
+export const parseTs = (file: string, data: ?string, additionalPlugins: string[] = []): ParseResult =>
+  parse(file, data, {plugins: [...plugins, ...additionalPlugins, 'typescript']});


Allow customization of the plugin is great 👍 , but I think I would make it more "optional" by providing a solid default set of plugins but allow override (customization might mean add or remove plugins). I think this way it provides the least interruption while adding flexibility.

i provided a more solid list of default plugins. (for feature i see used more regularly).
With spreading those plugins together the consumer is able to add addtional plugins.
But i don't see why anyone would want to remove plugins, this would require a different implementation. maybe like: use the alternativePlugins instead of plugins if provided.

README.md

connectdotz · 2020-02-08T17:50:49Z

src/__tests__/parsers/babel_parser_plugin.test.js

+    parse('fileName.ts', codeTobeParsed, false, additionalPlugins);
+    expect(spy).toBeCalledWith(codeTobeParsed, parserConfigTypescript);
+  });
+});


not sure if I missed it but I didn't see any additional fixture for testing plugin specific syntax?

no you didn't missed them, i have not added any yet.
i thought about this topic, normally you would only test your package, not the 3rd party depdenencies the project is using.
the plugin features should have already been tested in the babel project.

i mean we could not even fix a issue, if some tests were red, since its not within this project.

I see where you are coming from. I think I am less concerned about "if babel implemented the plugin correctly" but "if the plugin list we passed in has the effect we anticipated"...

I played with the new babel-parser_plugin.test.js to try to answer my own question:

comment out all additional plugins, it passed; ok, maybe all of our code fixures/snippets are just standard js that requires no additional plugin...?

passed in a garbage plugin, it still passed. ok, maybe babel load plugin like a best-effort...?

I then created some optional chaining code-snippet with empty additional plugin list, thinking for sure it should fail without passing the correct plugin, right? But surprise surprise, it still passed.

At this point, I feel there is something I don't understand or misunderstand about this plugins thing... maybe it is reading the babel.config.js locally? does that mean it will be impacted by users' env as well once we deployed, which might be good, or bad... How do tools like vscode-jest or yours decide what plugin list to pass in? If they can't decide then should all users now need to pass in their own list, even if they don't use babel? With the current implementation (minimal default), how disruptive the next release will be? Do we really need to get tangled with the rather complex user babel config/env?

Anyway, tests mean to surface issues, and I think without the simple additional tests I would not have discovered the unexpected behaviors and questions... in addition to tests, we should answer the questions above to make sure we fully understand what the community and users might face with these changes... thoughts?

i don't have an answer to your questions yet but i appreciate that you are trying to understand it properly. so please don't get my comment wrong.

just one side thought, which might be interesting (in terms of testing)

majestic doesn't seem to take this topic that seriously:
https://github.com/Raathigesh/majestic/blob/master/server/services/ast/parser.ts#L10

they just use 3 plugins => "jsx", "classProperties", "optionalChaining"

Hey @connectdotz. I had a look at this too and think that I've made some headway. Have a look at this babel documentation that I found helpful (I thought I knew what the plugins did by the names - I wasn't always right). That page will give you some useful snippets to test with later on.

Experimenting with babel-parser_plugin.test.js:
Hopefully I'm not pointing out the obvious, but this file is aimed at testing whether the correct plugin config is passed to the parser rather than whether that configured parser can parse the code correctly. You can see this by the fact that the "code" it is asked to parse is on line 26:

const codeTobeParsed = 'const bla = 1';

which as you can see is pretty simple JS and is not testing any of the experimental language features that the plugins are supposed to be supporting. Which is probably why when you removed all the plugins, it still parsed correctly and the tests passed.

When I extended that line with snippets from the Babel documentation like this:

const codeTobeParsed = ` //classPrivateProperties class A { #b = 1; } //classPrivateMethods class B { #c() {} } //doExpressions var a = do { if (true) { 'hi'; } }; `;

I got test failures when the appropriate plugins were not enabled. Hope that helps.

exacly @rossknudsen this test was only to verfiy if the addtionalPlugins have been merged with the default plugins correctly.

since i have added this feature i thought it make sense to test it.

also did some research on this topic.

babel-parser_plugin.test.js is a good place to test the babel plugins.

i found that for example ObjectRestSpread works even if no plugin is activated. While classPrivateProperties or doExpressions do only work with plugins activated.

that is probably because some proposal are already on a later stage and work even without explicitly activating them.

ObjectRestSpread and OptionalChaining are already in stage 4.

does this answer your concerns @connectdotz ?

@firsttris yes it does! thanks for doing the research 👍

I also dig into the babel-parser code today and found that it checked for only a small set (~11) of plugins, such as logicalAssignment, classProperties, and the document did mention

The latest ECMAScript version enabled by default (ES2017).

This is great and does eliminate my biggest concern. Obviously, as the language evolves the extra plugin list will too, but as long as we keep the babel-parser up to date we should be able to support the majority of the use cases.

Another take away from babel-parser code: The logic to recognize and process these non-standard language features are already in the parser, the plugin strings are just like flags to allow processing the corresponding syntax if encountered in the content-to-parse.

For jest-editor-support, I don't see much downside to allow the parser to parse as much as it can. If the users choose to constraint it for their specific use case, which should be rare, they can certainly provide its plugin list to narrow and override the default.

there are 2 plugins require additional flags, if there is really no obvious winner (more commonly used), then whichever flag is fine, provided we document it so users know how to override it if need to.

There is no point to pass in "new" plugin because the babel-parser we packaged will not understand them anyway...

package.json

Co-Authored-By: Sean Poulter <[email protected]>

rossknudsen · 2020-02-26T08:31:25Z

@firsttris yeah it's because the JS files have Flow typings in them. TS compiler doesn't understand them. Which is why we're using Babel to transpile, as it understands everything.

So the tsconfig is really only being used to typecheck the TS files and we want it to ignore the JS files.

firsttris · 2020-02-27T10:41:56Z

hey @connectdotz & jest-community, i did my best to implement your suggestions and change requests.

If you have additional change request please let me know. once i find time between my worktime, i will do my best to implement them.

really want to continue extend that parser with test.each support, i see many projects heavely using this and our addons are not supporting this feature.

From the user's perspective, that's a bug.
jest-community/vscode-jest#427
firsttris/vscode-jest-runner#55

connectdotz · 2020-02-27T16:24:36Z

hey @firsttris sorry for the long delay, was really busy this month, I will try to take a look this weekend, thanks for being patient.

connectdotz

hey @firsttris not sure if you are still interested in this PR or maybe we both are waiting for the other to take the next step...?

I see the major blocker is in how we invoke babel-parser, i.e. the options, as well as how we customize it... see the inline comments.

This change will most likely impact the README, tests, and maybe others... let me know when you are happy with the changes, I will do a final review then

Sorry for such a long delay, hope you haven't given up...

CHANGELOG.md

connectdotz · 2020-04-22T15:47:09Z

src/parsers/babel_parser.js

+export const plugins = [
+  'jsx',
+  'classProperties',
+  'exportDefaultFrom',
+  'logicalAssignment',
+  'nullishCoalescingOperator',
+  'objectRestSpread',
+  'optionalChaining',
+  'topLevelAwait',
+];
+
+// Its not possible to use the parser with flow and typescript active at the same time
+export const parseJs = (file: string, data: ?string, additionalPlugins: string[] = []): ParseResult =>
+  parse(file, data, {plugins: [...plugins, ...additionalPlugins, 'flow']});
+export const parseTs = (file: string, data: ?string, additionalPlugins: string[] = []): ParseResult =>
+  parse(file, data, {plugins: [...plugins, ...additionalPlugins, 'typescript']});


giving the plugins here are just "flags" for the babel-parser, I suggest we add all of them, you can reference their list here.

Once we enabled everything, there is no point to add "additional"... the customization should be very rare, if any, and it could be simply an override.

it was not possible to add every plugin, but i added almost every plugin. i removed the ones which caused conflicts.

i wonder if adding all those plugins is even necessary

firsttris · 2020-04-22T16:11:02Z

hey @connectdotz to be honest is was not totally clear for me what tasks where missing to complete this PR.
the PR is really big, and its seems not really transparent in general on github.

to summarize

the open tasks are:

we remove the option to pass additional plugins, instead we include all plugins
update readme accordingly
update changelog (add author)

do you see any additional changes to be done @connectdotz ?
(sorry if i missed something the PR is so big)

connectdotz · 2020-04-23T13:35:26Z

@firsttris your summary is right, I will take another look when you are done...

firsttris · 2020-05-19T19:13:09Z

did it

but travis now says

The command "eval yarn --frozen-lockfile " failed 3 times.
The command "yarn --frozen-lockfile" failed and exited with 1 during .

what can i do about it?

rossknudsen · 2020-05-20T06:19:58Z

This is the error:

error [email protected]: The engine "node" is incompatible with this module. Expected version ">=10". Got "8.17.0"

I think you need to change the node version to 10 in the Travis config. I think I had to do that in my PR.

firsttris · 2020-05-25T18:21:11Z

thx @rossknudsen that worked.

just coveralls is complaining now.

connectdotz

@firsttris thanks for the update, we are almost there! a few non-blocking document comments, and a question about the plugin list.

Don't worry about coverage complaint, we should be good to go once we resolve the issues below

README.md

connectdotz · 2020-05-26T18:05:03Z

README.md

+- filePath = Path to the file you want to parse.
+- serializedData = Serialized data, will be used instead of the filePath if available (optional).
+- strictMode = If this option is activated the parser throws an exception if the filetype is not detected, defaults to false.
+


great way to start documenting the API 👍 , can we throw in an example, I am a big fan of "learn-by-example'

would be best to create an example with typescript and to link it in the readme (maybe follow up PR)

package.json

connectdotz · 2020-05-26T18:22:04Z

src/parsers/babel_parser.js

+export const plugins = [
+  'asyncGenerators',
+  'bigInt',
+  'classPrivateMethods',
+  'classPrivateProperties',
+  'classProperties',
+  'doExpressions',
+  'dynamicImport',
+  'estree',
+  'exportDefaultFrom',
+  'exportNamespaceFrom', // deprecated
+  'functionBind',
+  'functionSent',
+  'importMeta',
+  'jsx',
+  'logicalAssignment',
+  'nullishCoalescingOperator',
+  'numericSeparator',
+  'objectRestSpread',
+  'optionalCatchBinding',
+  'optionalChaining',
+  'partialApplication',
+  'throwExpressions',
+  'topLevelAwait',
+];


looks like there is something missing from the babel plugin list, is it intentional?

there are conflicts between various plugins, tried to include all and removed the ones which caused conflicts

connectdotz · 2020-05-27T16:05:05Z

hmm... on the 2nd thought, I think this is a good enough start, I can fix the remaining minor issues in another PR so we can free you to move on to the original thing you wanted to fix:

if you are happy with this PR, i would proceed update the jest parser to new features like test.each.

@firsttris if there is anything you want to add I can wait, otherwise I can merge it tonight.

firsttris · 2020-05-28T12:22:46Z

@connectdotz i'm rdy

connectdotz · 2020-05-29T00:37:19Z

@firsttris thank you for your hard work 🙏 and sorry it took so long.

seanpoulter · 2020-05-29T02:05:30Z

Yay! Thank you Tristan! 🥇

connectdotz · 2020-05-31T02:46:07Z

@firsttris while testing the new typescript babel_parser, noticed a discrepancy vs the old typescript parser. While the old parser does not return the right "name" for the jest.each, it still returns the test block. The new parser completely skipped the block...

If you already started to work on the jest.each then I will just wait, otherwise, I can take a look to at least closing the parity...

firsttris · 2020-05-31T06:52:39Z

Have not yet started looking in to this topic.

connectdotz · 2020-06-01T20:44:57Z

ok, I did some quick fix at #47, would love to have you review it.

firsttris requested review from seanpoulter and connectdotz January 24, 2020 19:16

seanpoulter requested review from orta and stephtr January 25, 2020 06:14

stephtr reviewed Jan 25, 2020

View reviewed changes

fixtures/parser_tests.js Show resolved Hide resolved

src/parsers/babel_parser.js Outdated Show resolved Hide resolved

src/parsers/babel_parser.js Outdated Show resolved Hide resolved

rossknudsen mentioned this pull request Jan 26, 2020

Typescript conversion #32

Closed

seanpoulter approved these changes Jan 31, 2020

View reviewed changes

connectdotz requested changes Feb 1, 2020

View reviewed changes

src/parsers/babel_parser.js Outdated Show resolved Hide resolved

src/parsers/index.js Outdated Show resolved Hide resolved

firsttris commented Feb 3, 2020

View reviewed changes

src/parsers/index.js Outdated Show resolved Hide resolved

connectdotz requested changes Feb 8, 2020

View reviewed changes

rossknudsen reviewed Feb 10, 2020

View reviewed changes

package.json Show resolved Hide resolved

firsttris and others added 11 commits February 25, 2020 20:10

update babylon to babel/parser 7.x

74f2805

fix snapshot.test and typing

785f4be

update changelog

9abd415

remove @flow from parser_tests.js

fab6084

revert import in parser_tests.js

b3479ff

increase test coverage

c49d4a2

remove typescript

88f0260

make options optional

348ed8d

add addtional plugins

6c47166

Update CHANGELOG.md

97b37e8

Co-Authored-By: Sean Poulter <[email protected]>

fix changelog

5ffe53e

extend default plugin list

b907c59

rossknudsen mentioned this pull request Apr 15, 2020

Parametrized tests handling kavod-io/vscode-jest-test-adapter#26

Closed

seanpoulter self-requested a review April 15, 2020 18:19

seanpoulter mentioned this pull request Apr 15, 2020

it.each node range doesn't include callback body #37

Open

connectdotz requested changes Apr 22, 2020

View reviewed changes

This was referenced Apr 29, 2020

Concatenation in test case description causes test loading fail kavod-io/vscode-jest-test-adapter#43

Closed

Using function as module factory in jest.mock causes test loading fail kavod-io/vscode-jest-test-adapter#44

Closed

firsttris added 2 commits May 19, 2020 20:38

merge master

a3f9c00

update plugin list

5fd8d51

increadse node_js to 10 in travis.yml

e29beec

connectdotz reviewed May 26, 2020

View reviewed changes

firsttris added 2 commits May 27, 2020 20:10

update readme

22be841

fix prettier

3e61670

connectdotz merged commit 5e122c5 into jest-community:master May 29, 2020

firsttris mentioned this pull request Nov 26, 2023

add link to vscode-jest-runner to README jest-community/awesome-jest#132

Merged

update babylon to babel/parser 7.x #30

update babylon to babel/parser 7.x #30

Conversation

firsttris commented Jan 24, 2020 • edited Loading

stephtr left a comment

Choose a reason for hiding this comment

connectdotz commented Jan 26, 2020 • edited Loading

firsttris commented Jan 26, 2020 • edited Loading

connectdotz commented Jan 26, 2020 • edited Loading

stephtr commented Jan 26, 2020

seanpoulter left a comment

Choose a reason for hiding this comment

seanpoulter commented Jan 31, 2020

connectdotz left a comment

Choose a reason for hiding this comment

firsttris commented Feb 2, 2020 • edited Loading

firsttris commented Feb 2, 2020

connectdotz commented Feb 3, 2020

connectdotz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firsttris Feb 25, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firsttris Feb 9, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firsttris Feb 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firsttris Feb 25, 2020 • edited Loading

Choose a reason for hiding this comment

connectdotz Mar 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rossknudsen commented Feb 26, 2020 • edited Loading

firsttris commented Feb 27, 2020

connectdotz commented Feb 27, 2020

connectdotz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firsttris commented Apr 22, 2020 • edited Loading

connectdotz commented Apr 23, 2020

firsttris commented May 19, 2020

rossknudsen commented May 20, 2020

firsttris commented May 25, 2020

connectdotz left a comment

Choose a reason for hiding this comment

connectdotz May 26, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

connectdotz commented May 27, 2020

firsttris commented May 28, 2020

connectdotz commented May 29, 2020

seanpoulter commented May 29, 2020

connectdotz commented May 31, 2020

firsttris commented May 31, 2020

connectdotz commented Jun 1, 2020

firsttris commented Jan 24, 2020 •

edited

Loading

connectdotz commented Jan 26, 2020 •

edited

Loading

firsttris commented Jan 26, 2020 •

edited

Loading

connectdotz commented Jan 26, 2020 •

edited

Loading

firsttris commented Feb 2, 2020 •

edited

Loading

firsttris Feb 25, 2020 •

edited

Loading

firsttris Feb 9, 2020 •

edited

Loading

firsttris Feb 11, 2020 •

edited

Loading

firsttris Feb 25, 2020 •

edited

Loading

connectdotz Mar 5, 2020 •

edited

Loading

rossknudsen commented Feb 26, 2020 •

edited

Loading

firsttris commented Apr 22, 2020 •

edited

Loading

connectdotz May 26, 2020 •

edited

Loading