Like Transforms but for preprocessing instead of postprocessing

[MangelMaxime]

Is your feature request related to a problem? Please describe.

In some languages, it is possible to embed documentation inside of a script file so user can benefit from compiler type checking when writing their documentation/blog post.

For example, in F# you can write a "real F# script" like that:

// This is a code block
let value = "Hello World"

(***
# This a documentation comment

Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Donec euismod, nisl eget consectetur consectetur, nisl nunc egestas nisi, eu consectetur nunc nisi euismod nunc.

***)

// More code block below

let add (a : int) (b : int) = a + b

and once transformed by a tool/plugin it could become something equivalent to that:

```fs
// This is a code block
let value = "Hello World"
```

# This a documentation comment

Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Donec euismod, nisl eget consectetur consectetur, nisl nunc egestas nisi, eu consectetur nunc nisi euismod nunc.

```fs
// More code block below

let add (a : int) (b : int) = a + b
```

Describe the solution you'd like

I would like the ability to pre-process a file to transform it into markdown for example, and then have the standard eleventy pipeline kicks in to benefit from all the features.

In the past, I opened a discussion to ask for help on how to workaround front-matter/plugins limitations. #2505

But after, coming back to it, I do think that this could be a more generalised need in order to support literate programming for languages that supports it.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

I tried to use the plugin system but I found 2 problems:

1. Currently, we can't set several front-matter delimiter meaning if I set it to (*** and ***) to have valid F# code I need to use these delimiters even in the markdown files, which leads to bad tooling experience for the user
2. The nunjucks instructions are not processed correctly

Another, alternative I can think of would be to have a tool running before/side by side with eleventy which would transorm all script1.fsx into a script1.fsx.md so Elventy would see the script1.fsx.md as a normal file.

And I would probably be able to add *.fsx.md to the .gitignore configuration file.

However, looks more like a hack than a transparent user experience.

Additional context
Add any other context or screenshots about the feature request here.

[Snapstromegon]

I really like this request and my first thought would be something like transforms but for preprocessing instead of postprocessing files.

If something like this is implemented, we should clearly document that imported things after the post processing stage (like a nunjucks import) will not be affected by this (or at least I think it shouldn't be affected).

Also this type of "scripts with docs" is also really common in other filetypes like Jupyter Notebooks, MatLab files, or others. Supporting these documentation generating purposes could be really interesting and open up 11ty to other usecases.

[j-f1]

I think the best solution here would be to expose an API that provides the equivalent of renderTemplate, but built-in and within the render callback, or this.defaultRenderer (but with the ability to pick an arbitrary renderer), to allow custom languages to have more control over how the formatting is applied. For example, it might make sense to run each comment block of a literate file through the template language separately, so that anything that happens to be both a Nunjucks command and valid F# code, for example, doesn’t get messed up.

[zachleat]

Considerations for this feature:

1. How does it affect rawInput? Should it come before or after? addTransform and addPlugin run order has changed in 1.0.0 ? #2044 1206
2. Seems also related to More extensible template syntax chaining. #3298 (arbitrary and extensible syntax chaining)

Moving this to the enhancement queue.

This repository is now using lodash style issue management for enhancements. This means enhancement issues will now be closed instead of leaving them open.

View the enhancement backlog here. Don’t forget to upvote the top comment with 👍!

[MangelMaxime]

@zachleat Thanks for looking into it.

1. How does it affect rawInput? Should it come before or after? addTransform and addPlugin run order has changed in 1.0.0 ? #2044 1206

I believe this should come first because the goal is to prepare the input and then forward it to a "default" template engine.

1. Read input file
2. Apply a transformation to the input file to goes from F# syntax to Markdown
3. Process the result of step 2 as if the original file was a markdown file on the disk

Regarding both the following comments:

2. Seems also related to More extensible template syntax chaining. #3298 (arbitrary and extensible syntax chaining
this.defaultRenderer (but with the ability to pick an arbitrary renderer), to allow custom languages to have more control over how the formatting is applied.

Since I opened this issue, I was able to improve the situation in my code. In the past, I had to re-instantiate the differents TemplateEngine that I need to process the file (Markdown and Nunjuck). However, by using the EleventyRenderPlugin I found out that I could access the eleventy instances thanks to eleventyConfig.getFilter("renderTemplate").

const markdownContent = transformFsxToMarkdown(content);

const markdownText = await eleventyConfig.getFilter("renderTemplate")(markdownContent, "md");

// Replace " with " because some markdown renderers escape the quotes
// and it breaks the nunjucks compilation
const fixedMarkdownText = markdownText.replace(/"/g, '"');

return await eleventyConfig.getFilter("renderTemplate")(fixedMarkdownText, "njk");

This makes the code much easier to work with and a little bit less brittle. I still had some stuff breaking when upgrading to the latest alpha of Eleventy but it was probably because I don't always provide the actual page data/context or fake it correctly depending on where I am calling these functions.

[zachleat]

I believe #3298 solves this (with the approach @j-f1 suggested). For folks that want this, please follow along to #3298 (which is coming to 3.0 soon).

With that issue shipping soon and your comments regarding finding a workaround @MangelMaxime, I don’t think we need this on the enhancement queue any more either?

[zachleat]

#3298 is shipping with 3.0.0-alpha.11

[MangelMaxime]

@zachleat I need to give it a try but from what I understand yes.

[zachleat]

I do also want to cross-link one of the highest ranking issues on the enhancement queue (#188) too