PoC for code-snippets-in-markdown system

[kianenigma]

Recall:

• Both the master tutorial's (*THE* Polkadot SDK Tutorial #3) written code and reference docs (Tracking Issue for Reference Docs #4) will have code snippets.
• None of these should to be hardcoded and all must come from valid sources that can be checked for validity, such as a git permalink.

We need a system to build this, using Gatsby.

Note that some more details might change as we flesh out how the master tutorial is being built. Perhaps, if we use diff files to denote each step of the tutorial, the snippets must come not from a permalink, but instead from a diff file.

Moreover, we should also explore embedding pieces from docs.rs.

[kianenigma]

Did a bit of research on this. Looked for some rust tutorials to see if any are using a similar system, and can't find any. Some examples:

• https://github.com/stevedonovan/gentle-intro/blob/master/src/3-filesystem.md
• https://raw.githubusercontent.com/amethyst/rustrogueliketutorial/master/book/src/chapter_41.md
• https://github.com/jsoverson/node-to-rust/blob/master/book/chapters/chapter-10-traits.adoc

These are all md-books style stuff, and are all copy-past-ing code around.

[sacha-l]

I think @jonnysmillie may have some insight here based on stuff we were working on a while ago.

[hummusonrails]

There is definitely the possibility of extracting lines of code out of files from github repos and inserting them into markdown documents. Something like this came up in a quick search for Gatsby solutions, specifically, albeit it's not well maintained, but there are probably other implementations.

I think what is novel in what you propose is the moving forward by applying diffs to each step in the tutorial. Perhaps we can write the first iteration of this tutorial with static steps that take the reader through the various code stages making it clear that at, imagining a 5 step process, the code will not work at steps 1-4, and only once you've finished the tutorial at step 5, will the code work.

In regards, to making a CI/CD process for incomplete code snippets, that's more of a challenge. Perhaps we design a GH Action that pulls the code snippet from each stage, applies the diffs to create a complete code example, and then runs the tests against that. In other words, while I think it might be more difficult to render in the markdown documents the diff procedure you outline, we can probably do that for the CI/CD workflow in the snippets repo.

[kianenigma]

I had a good chat about this with @sam0x17 and we identified that all existing solutions seem to be "permalinks on steroids", meaning that end of the day, you are referencing some lines of code.

This is great, and let's already explore the feasibility.

But, as the saying goes, if you are a hammer, you see everything as nails. In the case of me and @sam0x17, we have a tendency to solve all things with Rust/Macro solutions 😅.

The idea would be to write a rust-markdown-processor that would parse some markdown, and parse some rust-files, and replace a special annotation in the markdown files with the associated Rust item. The cool thing about this is that you can reference any item, not just lines of code. In other words, a snippet could import a function, with its entire signature, into the markdown file.

This has an edge over permalink, as you don't need to worry about breaking the line numbers. Imagine if a big file is used in many places, and we suddenly move everything to one line below because of adding a new import 🤦

Before doing this, we should ensure it does not exist.

This is really only possible with Rust, because of syn crate that can read and understand the Rust syntax tree.

[hummusonrails]

But, as the saying goes, if you are a hammer, you see everything as nails. In the case of me and @sam0x17, we have a tendency to solve all things with Rust/Macro solutions 😅.

I think this is a really interesting idea, but I'm focusing in on this sentence from your comment above intentionally.

I want to propose a two-phased process here:

First, create a quality master tutorial that lives on an easy to navigate usable doc site.
Second, refactor that initial go with improved devex like what you mention.

I think you can approach the docs creation process as an engineer or as a technical writer. The technical writer wants to produce the best doc possible, and the engineer wants to create the best solution for implementation possible. Both can be done, of course as they should be, yet I think some prioritization is necessary here.

It is a well known solution to use line numbers in markdown docs sites to pull in code from GitHub. It is definitely more brittle as you identify, but because it is so well known it'll allow us to move faster towards a finished v1 product before the next PBA, which I believe is your deadline, since there are so many ways to implement it.

At the same time, @sam0x17 and yourself and others (maybe even me, I really like the idea!) can work on building the new rust-markdown-processor as a separate crate.

In short, I think it's important to identify the minimal viable product for the v1 of the master tutorial and docs, then aim to achieve that before PBA. Then, we need to identify devex and other improvements we want to achieve in iterations of that v1 and a priority order of which one comes first. Personally, I'd recommend this markdown processor idea to be one of the early improvements on the first release.

[hummusonrails]

It seems @sam0x17 has made my last comment irrelevant by building this thing over the weekend. The idea I stated still stands but in practice is no longer relevant for this conversation :)

[sam0x17]

So where we stand right now is I have an unpublished feature for docify that can take in a single markdown file or a directory (recursively) containing many markdown files and it will process them, which involves going through each one, looking for a comment like <!-- docify::embed!("some/path.rs", some_ident) --> and then expands such comments to codeblocks containing the relevant example.

my vision for how this would be used was:

• you make a directory somewhere in your project that has the markdown files with <!-- docify::embed!(..) --> tags referring to other places in the project.
• you make a non-git-tracked directory that is your output directory
• you place docify::compile_markdown_dir!("input/dir", "output/dir"); somewhere in your project with a #[cfg(doc)] gate on it. This will cause the output dir to generate at docs compile time when you run cargo doc
• Depending on how docs.rs/etc and rust docs handles this, you possibly might be able to link directly to the generated markdown files in various places in the rust docs. I haven't tried it, but I don't see why it wouldn't work. If that works then docs.rs could work as a way to publish all the tutorials.

Based on our discussion, the following would be desirable:

• expose this as a simple-to-use CLI (my vision as I outline above was this would all run as part of the monorepo's cargo doc at compile-time, but CLI is fine too and I have no objections either way)
• add the option to have it output github embed links instead of codeblocks
• add the option to specify specific commit hashes in the embed tag (this would involve shelling out to git but should be fairly straightforward). It would mean git becomes a compile-time dependency, though that is probably uncontroversial.

[hummusonrails]

@sam0x17 This sounds fantastic, and super helpful.

[sam0x17]

btw, this feature (markdown dirs) is live as of late last week. If anyone encounters any issues let me know!

[kianenigma]

This is issue is in itself closed now as @sam0x17 built docify. If more features needed, we will communicate elsewhere.