generate test cases from our styleguide's README.md

[justjake]

WORK IN PROGRESS

1. get all the code blocks from the styleguide README.md
2. split out examples based on // bad, // good, // best comments
3. prepend common setup code that occurs before each delimiters to each
   example
4. test each example:
   • all examples must be valid Javascript code according to ESLint using
     our config.
   • good examples must lint without errors or warnings
   • best examples must lint without errors or warnings
   • bad examples must fail linting

In the next commits, I plan to edit README.md to reduce the number of examples with irregular delimiters like // good - bla bla bla bla bla

[ljharb]

Sweet, this LGTM

[mmwtsn]

This is great.

If that functionality is at all reusable being able to extract JavaScript in GitHub-flavored Markdown blocks from a project's README to lint would be a useful npm module to have outside this repo.

[justjake]

@mmwtsn I'll see what ends up general enough to package. I'm considering splitting the README up into multiple documents with more annotation around the test cases and slightly easier formatting. I'm also considering writing some python-doctest inspired stuff to verify // -> [1, 2] style value annotations

[justjake]

alright i have a format for styleguide sections that I'm pretty happy with; the main feature is no more manually tracking rule numbers and tables of contents.

Here's a demo of building from the easy section format to the existing markdown format
https://gist.github.com/justjake/0aaf202f67a7e102b0c5

[ariporad]

Hey, @justjake, I've been looking at this PR for a while, and I think it's really awesome. I was wondering what the status on it was?

I'm going to go look through it now, and see if I could figure out a way to help (if you'd like that).

Thanks again!

[justjake]

@ariporad sure if you're feeling generous. I worked on this a couple weekends in a row when I had no plans. I got a bit carried away with a side-project to rewrite the README as a bunch of seperate markdown documents with a more reasonable format -- I hate having to maintain indentation and manually ordering things.

So, there are two things going on in this PR really: I started with the test extraction stuff, but then I moved on to Markdown AST manipulation and generating .md documents from other .md documents.

I'm not actively working on this idea any more because styleguide meddling is hard to justify to myself over my other projects. I still think re-organizing the guide and producing tests from the styleguide itself is a worthwhile endeavor.

If I were to start working on this branch again, I would first write a script to automatically create the section markdown files from the existing (master) README.md since there have been many changes to the master styleguide.

Then, I would clean up the styleguide representational classes to make extracting test cases easier.
Finally, I would use the styleguide generation classes to re-write https://github.com/airbnb/javascript/pull/480/files#diff-92afd690ee7051d5486860c89df391f5R1 so there's just one way of parsing markdowns into styleguide classes and working with them.

[ariporad]

Hmm...

If it's OK with you, I'm going to try a simpler way of combining README files, so more focus can be done on the test cases. If possible, I'd even like to auto-generate the .eslintrc, as opposed to the tests.

[justjake]

👍

[ljharb]

When we're ready to reattempt this, we should use eslint-plugin-markdown for it.