feat: Import Code Block from source code

[dickermoshe]

When working on the Dart ORM package, Drift (https://github.com/simolus3/drift), our founder created a static site generator too.
It's not as nice as this one, but it had a feature that I have never seen in any other static site generator.
It would be awesome if this package had such a feature.

One of the annoying parts of documenting a package is making sure that example code is up to date.
If changes are made to the project that changes some of the syntax, it's quite easy to forget to update the text in the markdown.

However, imagine if we could keep all the code in .dart (or anything else .py,.js etc.) and import the code blocks directly.

For example, in our source documentation, we have sections that look like this:

/manager.md

...read rows from the table or watch for changes.

{% include "blocks/snippet" snippets = snippets name = 'manager_select' %}

The manager provides a really easy to use API...

When building, the site generator goes through all the dart files in blocks/snippet, looks for a snippet named manager_select and loads the code into the markdown resulting in docs like this:

...read rows from the table or watch for changes.

\```dart
Future<void> selectTodoItems() async {
  ///. .. source code from a dart file
}
\```

The manager provides a really easy to use API...

Marking an area with a code snippet looks something like this:

// #docregion manager_select
Future<void> selectTodoItems() async {
 ///...
}
// #enddocregion manager_select

This could work with any lang, parsing the file extension and putting the correct lang code on top of the multi-line code snippet.

.py == ```python
.dart == ```dart
etc. 

The primary benefit of this is that code is not being written in Markdown files, it being written in source code files, where tests could be ran and code can be shared.

Imagine if you could use your test code as example code!

If interest for this feature is expressed, I will gladly put some time into it

[matthew-carroll]

Thanks for filing this @dickermoshe - CC @angelosilvestre

I've talked with Angelo a number of times about situations similar to the one you described. We've looked at a view different syntaxes that might accomplish the goal. @angelosilvestre can you mention the syntax details that we discussed most recently to mark areas of source code that should be documented?

I agree that copying code into docs is counterproductive, and it would be nice to automate that. I'd like for us all to consider the variety of use-cases associated with this and then come up with a syntax that covers all reasonable use-cases.

For example, consider the following desired goals:

• I want to show a code block with a few methods, but the methods are spread out across the codebase.
• I want to display a specific State object, but I only want the build() method displayed within the State class. I don't want to include all the other methods.
• I want to show a method, but it's a long method, so I want to hide certain areas within the method.

If we can figure out a reasonable syntax, and if we can figure out a way for this behavior to work regardless of where the static site sits compared to the source code, then I'd be happy to see this as a plugin included within static_shock.

[dickermoshe]

I don't have anything to add other than that this syntax should not be language specific at all.

[angelosilvestre]

I experimented with this a long time ago. It wasn't designed to be part of a static generator, so it only generates markdown for a single file and it was meant be used to generate step by step guides.

The syntax is like:

// magic_prefix:>step:{step number} {block description}
void myMethod() {}
// magic_prefix:<step:{step number}

Reference: https://github.com/angelosilvestre/code2docs

Currently, none of the @matthew-carroll's use cases are supported, but we can come up with a syntax that works for these use cases.

[matthew-carroll]

@angelosilvestre feel free to brainstorm anything that comes to mind. We can collect all the reasonable options and then pick whatever seems most versatile. Or perhaps we need a few different syntaxes for different use-cases.

[matthew-carroll]

CC @suragch in case you have thoughts on this, too.

[suragch]

Keeping documentation/tutorial code up-to-date and error-free is definitely a big need. In the past I've created unit tests or entire repos for the code used in a chapter or article, but it still required copying the code by hand. That would be amazing to be able to directly unit test the code that is included within an article.

I have no idea what the syntax would be to only show partial code snippets like the use cases Matt is talking about. Perhaps some sort of marker in the source code that correlates it with a tag in the doc code block.

I recall that Bob Nystrom was able to pull in code and test it when he wrote Crafting Interpreters. He discusses that here, but I don't understand how he did it.

[matthew-carroll]

It's probably a worthwhile exercise to check for any existing languages/tools/approaches to see what others have been able to accomplish. Obviously this isn't a new problem. I'll check out the link for Bob's approach. But also no need to limit ourselves to Flutter/Dart - an approach in a JS project, Ruby project, Python project would all probably inform the effort.

[dickermoshe]

Basic Snippet

A code snippet which just contains one section:

// @docregion snippet_name
void main(){
  print("Hello World");
}
// @enddocregion snippet_name

Snippet with segments

// @docregion snippet_name 1
void main(){
  print("Hello");
  // @enddocregion snippet_name 1
  print("Hidden");
  // @docregion snippet_name 2
  print("World");
}
// @enddocregion snippet_name 2

This snippet would be stitched together from all the segments

I agree, something that already exists were be much better.

[suragch]

I haven't studied these in great depth, but here are some solutions that seem to be trying to solve a similar problem for other SSGs:

Hexo

https://hexo.io/docs/tag-plugins#Include-Code

{% include_code [title] [lang:language] [from:line] [to:line] path/to/file %}

MkDocs

https://squidfunk.github.io/mkdocs-material/reference/code-blocks/#embedding-external-files
https://facelessuser.github.io/pymdown-extensions/extensions/snippets/#snippets-notation

Gatsby

https://www.gatsbyjs.com/plugins/gatsby-remark-embed-snippet/

[dickermoshe]

I would caution against any syntax that depends on line number.
Before any rebuilding of the docs, you would need to painstakingly go through each "include_code" and see if the lines changes.
It's not maintainable long term.

I think we should do Gatsby, however, I think we should not include line numbers at all.

[angelosilvestre]

I agree we should rely on line numbers. Also, I think we should use a magic prefix to make it easy to distinguish between syntax comments and regular comments. For example:

// shock: region snippet_name
void main(){}
// shock: endregion snippet_name

To show only a portion of a file, we could do something like the following:

// shock: region snippet_name
void main(){
  print('This line will be rendered');
  // shock: snippet_name hide 
  print('No lines will be displayed until we find a "show" directive');
  // shock: snippet_name show
  print('This line will be rendered');
}
// shock: endregion snippet_name

We could also declare multiple sections on the same block. That way, the docs could contain a block with parts of the code and another block with the full code. For example:

// shock: region partial_region
// shock: region full_region
void main(){
  print('This line will be rendered');
  // shock: partial_region hide 
  print('No lines will be displayed until we find a "show" directive');
  // shock: partial_region show
  print('This line will be rendered');
}
// shock: endregion partial_region
// shock: endregion full_region

If the user includes the region "partial_region", the following will be rendered:

void main(){
  print('This line will be rendered');
  print('This line will be rendered');
}

If the user includes the region "full_region", the following will be rendered:

void main(){
  print('This line will be rendered');
  print('No lines will be displayed until we find a "show" directive');
  print('This line will be rendered');
}

@dickermoshe @suragch Any thoughts on this syntax? Do any of you have another use-case that we should consider?

[matthew-carroll]

One random thought after looking at one of the samples that @suragch posted, I wonder if it would be productive to meld together the concept of line numbers with labels. We wouldn't use actual line numbers, because that's not maintainable. But we could use labels so that the documentation has a bit more control over what's included.

// @start: build
@override
Widget build(BuildContext context) {
  // @label: setup
  final thing = ...
  final other = ...

  // @label: tree
  return SuperEditor(
    ...
  );
}
// @end: build

Then various possible includes:

{# Includes the entire build method from @start to @end #}
{% source_code block:build %}

{# Includes lines after "setup" but before "tree" #}
{% source_code snippet:setup %}

While it has been requested that this syntax be language agnostic, I'm wondering if we should have generally useful agnostic syntax, but also some syntax that understands Dart and thereby can shorten requests:

{# Include the whole build() method by name %}
{% source_code dart method:SuperEditor.build %}

{# Include just the widget tree from the source code example above, because Dart knows where the method ends #}
{% source_code dart snippet:tree %}

[dickermoshe]

I love the syntax, except for one thing.

{# Includes lines after "setup" but before "tree" #}
{% source_code snippet:setup %}

This would mean that labels would have to be unique too, which would defeat the entire purpose.
You would need to name label snippet_name__label_name for it to be unique everywhere.
Reminds me of css.

I would propose this:

{# Includes lines after "setup" but before "tree" #}
{% source_code build:setup %}
where:
{% source_code snippet_name:label_name %}

Getting into analyzing dart would be a nice addition, but it adds lots of complexity.
Maybe it's worth working on the simple bits first.

[matthew-carroll]

This would mean that labels would have to be unique too, which would defeat the entire purpose.

@dickermoshe - I didn't follow this point. Can you elaborate on what you mean? Which part are you calling a "label" and why would uniqueness defeat the entire purpose?

[dickermoshe]

I see 2 benefits to @label

1. You don't need to write @end for snippets of a block.
2. Easier naming scheme for snippets of a block.

---

1.

// @start: section_name
final a = "ignored";
// @label: label_name1
final b = "included";
// @label: label_name2
final c = "ignored";
// @end: section_name

If we only want to include final b = "included"; we don't need to create an entire block.
A block here would require adding a start and end,.
Instead, we can just include a single line for it: label_name1.
We would then intelligently include all the code in between label_name1 & label_name2

The syntax you're proposing would work great with this!

which would defeat the entire purpose.

I was wrong about this

---

2.

Under the current syntax these will get quite verbose, we are repeating the words my_widget and another_widget everywhere. See the example below:

Example

// @start: my_widget
class MyWidget extends StatelessWidget {
  const MyWidget({super.key});
  // @start: my_widget_build
  @override
  Widget build(BuildContext context) {
    // @label: my_widget_setup
    final thing = ...
    final other = ...
  
    // @label: my_widget_tree
    return SuperEditor(
      ...
    );
  }
  // @end: my_widget_build
}
// @end: my_widget

// @start: another_widget
class AnotherWidget extends StatelessWidget {
  const AnotherWidget({super.key});
  // @start: another_widget_build
  @override
  Widget build(BuildContext context) {
     // @label: another_widget_setup
    final thing = ...
    final other = ...
  
    // @label: another_widget_tree
    return SuperEditor(
      ...
    );
  }
// @end: another_widget_build
}
// @end: another_widget

What I'm proposing is that nested blocks and labels are accessed via their parents.

// @start: my_widget
class MyWidget extends StatelessWidget {
  const MyWidget({super.key});
  // @start: build
  @override
  Widget build(BuildContext context) {
    // @label: setup
    final thing = ...
    final other = ...
  
    // @label: tree
    return SuperEditor(
      ...
    );
  }
  // @end: build
}
// @end: my_widget

The entire widget:

{% source_code my_widget %}

Just the build method:

{% source_code my_widget:build %}

Just the setup section of the build method:

{% source_code my_widget:setup %}

[matthew-carroll]

So @dickermoshe it sounds like the specific point you're making is that blocks should automatically nest. Is that right? If so, seems reasonable to me.