run doc example code as a test suite

[StefanKarpinski]

The examples in the wiki should all be exactly what the interpreter actually produces. We should automate comparing the output of the interpreter with what is in the wiki, thereby ensuring accuracy of all the examples and serving as an additional set of integration tests. One issue is when to clear variables and state. An obvious choice is to retain state within sections (or subsections, or whatever).

[JeffBezanson]

This seems like kind of an annoying project. Are you really going to do this? It seems like there would be better uses of your time.

[StefanKarpinski]

It would be a good thing to have at some point. Not urgent though. Also, I suspect, not all that hard to do.

[StefanKarpinski]

And maintaining the wiki when we change the language is kind of annoying too. This might be less work in the long run.

[ViralBShah]

Actually, a better way would be to have some annotations in the wiki, which can be extracted as code, run automatically within julia, and the real output used. Given the wiki is actually just a git repo makes this quite easy to pull off.

[StefanKarpinski]

I thought about that, but I think using it as a test suite is for the best. If the output changes, you really want to actually examine it and make sure that the surrounding text still makes sense and applies.

[JeffBezanson]

Doesn't seem like this can/should block 1.0. Reprioritize?

[StefanKarpinski]

No, it shouldn't. Running through them all by hand should be fine. Can automate later.

[jiahao]

This calls for champagne. 🍷

[ViralBShah]

+1