metadock

Templated documentation engine, powered by Jinja2 + marko.

Quick Intro

Using markdown (.md) as a common source format for rich text content, metadock allows you to define Jinja templated documents for various markdown docs that you'd like to format into a rich text document, e.g. Jira, Confluence, Agile, Gitlab, or a static website. You can then compile your markdown documents using context variables supplied via yaml content schematics.

A simple Metadock-enabled project might look something like this:

MyProject/
  - ...
  - <my project content>
  - ...
  - .metadock/
      - templated_documents/
          - gitlab_mr_template.md
      - content_schematics/
          - gitlab_mr__feature1.yml
          - gitlab_mr__otherfeature.yml
      - generated_documents/
          - gitlab_mr__feature1.md
          - gitlab_mr__feature1.html
          - gitlab_mr__otherfeature.md
          - gitlab_mr__otherfeature.html

The root of your project is expected to have a .metadock folder, which can be generated from the CLI using metadock init.

Basic CLI Usage

The metadock CLI, installed using pip install metadock, has 5 basic commands, spelled out in the help message:

usage: metadock [-h] [-p PROJECT_DIR] {init,validate,build,list,clean} ...

Generates and formats Jinja documentation templates from yaml sources.

positional arguments:
  {init,validate,build,list,clean}
                        Metadock command
    init                Initialize a new Metadock project in a folder which does not currently have one.
    validate            Validate the structure of an existing Metadock project.
    build               Build a Metadock project, rendering some or all documents.
    list                List all recognized documents which can be generated from a given selection.
    clean               Cleans the generated_documents directory for the Metadock project.

options:
  -h, --help            show this help message and exit
  -p PROJECT_DIR, --project-dir PROJECT_DIR
                        Project directory containing a .metadock directory.

Each of the commands supports a programmatic invocation from the metadock.Metadock class via a Python interface.

metadock init

• Description: Used to initialize a fresh Metadock project in a folder which does not currently have one.
• Usage: metadock [-p PROJECT_DIR] init
• Python interface:
  • Name: metadock.Metadock.init
  • Signature: (self, working_directory: Path | str = Path.cwd()) -> metadock.Metadock

metadock validate

• Description: Used to validate the structure of an existing Metadock project.
• Usage: metadock [-p PROJECT_DIR] validate
• Python interface:
  • Name: metadock.Metadock.validate
  • Signature: (self) -> metadock.engine.MetadockProjectValidationResult

metadock build

• Description: Used to build a Metadock project, rendering some or all documents.
• Usage: metadock [-p PROJECT_DIR] build [-s SCHEMATIC_GLOBS [SCHEMATIC_GLOBS ...]] [-t TEMPLATE_GLOBS [TEMPLATE_GLOBS ...]]
• Python interface:
  • Name: metadock.Metadock.build
  • Signature: "(self, schematic_globs: list[str] = [], template_globs: list[str] = []) -> metadock.engine.MetadockProjectBuildResult"

metadock list

• Description: Used to list all recognized documents which can be generated from a given selection.
• Usage: metadock [-p PROJECT_DIR] list [-s SCHEMATIC_GLOBS [SCHEMATIC_GLOBS ...]] [-t TEMPLATE_GLOBS [TEMPLATE_GLOBS ...]]
• Python interface:
  • Name: metadock.Metadock.list
  • Signature: (self, schematic_globs: list[str] = [], template_globs: list[str] = []) -> list[str]

metadock clean

• Description: Used to clean the generated_documents directory for the Metadock project.
• Usage: metadock [-p PROJECT_DIR] clean
• Python interface:
  • Name: metadock.Metadock.clean
  • Signature: (self) -> None

Example Usage

In the example above, we can imagine the content of our template, gitlab_mr_template.md, to look something like this:

{%- set jira_project_name = jira.get('project_name') -%}
{%- set jira_project_id = jira.get('project_id') -%}
{%- set jira_ticket_num = jira.get('ticket_num') -%}
{%- set jira_ticket_id = jira_project_name ~ "-" ~ jira_ticket_num -%}
{%- set mr_summary = merge_request.get('summary') -%}
# [{{ jira_ticket_id }}] {{ mr_summary }}

Welcome to my MR. Some of the changes are listed below:

{% for change in merge_request.get('changes', []) -%}
{{ loop.index }}. {{ change }}{{ "\n" if not loop.last else "" }}
{%- endfor %}

{% if merge_request.get('breaking_changes') -%}
In addition to the changes above, there are also a few breaking changes introduced in this MR:

{% for breaking_change in merge_request.get('breaking_changes') -%}
- {{ breaking_change.get('summary') }}
  - **Affected downstream stakeholders**: {{ breaking_change.get('affected_downstream', [{'id': 'None'}]) | map(attribute='id') | join(", ") }}.
  - **Suggested remedy**: {{ breaking_change.get('suggested_remedy', 'None') }}{{ "\n" if not loop.last else "" }}
{%- endfor -%}
{%- endif %}

For more information, please check out the Jira ticket associated with this MR, {{ jira_ticket_id }}.

This is a very simple MR format which can easily be generalized to allow for quickly generating large sets of docs which meet the same format and style requirements. An example content schematic which could service this template could be in gitlab_mr__feature1.yml:

content_schematics:

- name: gitlab_mr__feature1
  template: gitlab_mr_template.md
  target_formats: [ md+html, md ]

  context:

    jira:
      project_name: "IGDP"
      project_id: "12001"
      ticket_num: "13"

    merge_request:
      summary: Adding software version as hard requirement for staging
      changes:
        - "Added software version to staging model."
        - "Added unit tests for valid software version, invalid software version, missing software version."
      breaking_changes:
        - summary: "Dropping all records which are missing software version."
          affected_downstream:
            - id: Service
              email: [email protected]
            - id: Analytics
              email: [email protected]
          suggested_remedy: |
            - Drop all records which are missing software version.
            - Add software version as a hard requirement for staging.

By invoking the CLI with metadock build, our template is compiled to look something like this, in a markdown file called generated_documents/gitlab_mr__feature1.md:

[IGDP-13] Adding software version as hard requirement for staging

Welcome to my MR. Some of the changes are listed below:

1. Added software version to staging model.
2. Added unit tests for valid software version, invalid software version, missing software version.

In addition to the changes above, there are also a few breaking changes introduced in this MR:

• Dropping all records which are missing software version.
  • Affected downstream stakeholders: Service, Analytics.
  • Suggested remedy:
    • Drop all records which are missing software version.
    • Add software version as a hard requirement for staging.

For more information, please check out the Jira ticket associated with this MR, IGDP-13.

Because the target_formats we chose included md+html and md, we also get an HTML rendering of the document for free, located at generated_documents/gitlab_mr__feature_1.html:

<h1>[IGDP-13] Adding software version as hard requirement for staging</h1>
<p>Welcome to my MR. Some of the changes are listed below:</p>
<ol>
<li>Added software version to staging model.</li>
<li>Added unit tests for valid software version, invalid software version, missing software version.</li>
</ol>
<p>In addition to the changes above, there are also a few breaking changes introduced in this MR:</p>
<ul>
<li>
Dropping all records which are missing software version.<ul>
<li><strong>Affected downstream stakeholders</strong>: Service, Analytics.</li>
<li><strong>Suggested remedy</strong>: Handle deletions manualy, using the software version column in the exposures to identify source records
which will be dropped, and drop them in the target environment after our change is deployed.</li>
</ul>
</li>
</ul>
<p>For more information, please check out the Jira ticket associated with this MR, IGDP-13.</p>

In a single content schematics yaml file, you can define any number of documents which should be generated (and to what formats) using a given template and context (context variables loaded into the Jinja parser).

The template key should be a relative path to a Jinja template file from the ./metadock/templated_documents directory. The name key should be a unique identifier for the generated document, and will compose the basename of the generated file.

The natively supported values for target_formats are:

• md+html:
  • Generates the given template, parses it into a markdown document, and then generates HTML from it.
• Anything else, e.g. txt, sql or py:
  • Generates the given template as plaintext, and adds the given string as a file extension, e.g. .txt, .sql or .py.

Code splitting with YAML imports

In order to keep your content schematics DRY, you can use YAML imports to split your content schematics into multiple YAML files. For example, if you have a set of content schematics responsible for laying out a "knowledge base" of services maintained by your team, you might have a YAML file for each service, e.g. services/airflow/google_forms_scrubber.yml and services/pipelines/user_interaction_data_pipeline.yml which separately model their respective service specifications.

A content schematic can import context from a specific YAML key in another YAML file by using the special import-key object, e.g.:

content_schematics:

- name: alerting_project_proposal
  template: airflow_project_proposal_template.md
  target_formats: [ md+html, md ]

  context:

    jira:

      # "block" syntax for importing a root-level key "IGDP"
      project:
        import: jira/projects.yml
        key: IGDP

    # "flow" syntax for importing a sub-key, "David_Sillman" inside "eng_identity"
    code_owners: 
      - { import: jira/identities.yml, key: eng_identity.David_Sillman }

    # "flow" syntax for importing a sub-key using a merge key ("<<"),
    <<: { import: team_contexts/data.yml, key: resources.alerting_channels }

    # "block" syntax for importing multiple subkeys from multiple files using a merge key,
    <<:
      - import: team_contexts/data_contacts.yml
        key: contacts.email
      - import: team_contexts/data_push_api.yml
        key: push_api.contracts

Note that all paths for the import field are relative to the content_schematics folder for the project. If you'd like to import the entire content of a file as context, you may omit the key field, e.g.:

content_schematics:

- name: confluence_docs_summary
  template: confluence/data_docs/confluence_docs_summary_template.md
  target_formats: [ md+html, md ]
  context:

    # "flow" syntax for a single whole-file import,
    all_contracts: { import: confluence/data_docs/contracts.yml }

    # "block" syntax for importing multiple whole files using a merge key,
    <<:
      - import: confluence/data_docs/projects.yml
      - import: confluence/data_docs/sources.yml

At the moment, no protection against cyclic dependencies are implemented (apart from a recursion depth exception which will likely be thrown before memory is consumed). Users are responsible for ensuring that their imports do not create cyclic dependencies.

Jinja Templating Helpers

In the Jinja templating context which is loaded for each templated document, there are a handful of helpful Jinja macros and filters which can be used to make formatting content easier. The macros and filters are segregated into 3 namespaces, documented below:

Global namespace

Jinja namespace for the global Metadock environment, including all global macros, filters, and namespaces.

Jinja macros

The following macros are available in the global namespace:

• debug
• ref

Jinja macro reference

Macro	Signature	Doc
debug	metadock.env.MetadockEnv.debug: (self, message: str) -> None	Prints a debug message to stdout, and returns an empty string. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("No changes!{{ debug('This is a debug message.') }}").render() This is a debug message. 'No changes!'
ref	metadock.env.MetadockEnv.ref: (self, document_name: str) -> str	Renders and inserts the content from a given generated document in a given Metadock project. >>> from metadock.env import MetadockEnv >>> from metadock.project import MetadockProject >>> env = MetadockEnv(...).jinja_environment() >>> project = MetadockProject() >>> env.from_string("{{ ref('my_generated_document') }}").render() 'Rendered contents of my_generated_document'

Jinja filters

The following filters are available in the global namespace:

• chain
• inline
• with_prefix
• with_suffix
• wrap
• zip

Jinja filter reference

Filter	Signature	Doc
chain	metadock.env.MetadockEnv.chain_filter: (self, iterables: Sequence[Iterable[Any]]) -> Iterable[Any]	Filter which flattens a sequence of iterables into a single iterable. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string('{{ {"first": 1, "second": 2}.items() | chain | join(" ") }}').render() 'first 1 second 2'
inline	metadock.env.MetadockEnv.inline_filter: (self, value: str) -> str	Filter which inlines a string by replacing all newlines with spaces, and all double spaces with single spaces. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ 'This is a multi-line string.\nThis is the second line.\nAnd the third.' | inline }}").render() 'This is a multi-line string. This is the second line. And the third.'
with_prefix	metadock.env.MetadockEnv.with_prefix_filter: (self, value: str, prefix: str, sep: str = '') -> str	Filter which prepends a prefix to a string, with an optional separator. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ 'This is a string.' | with_prefix('Prefix') }}").render() 'PrefixThis is a string.' >>> env.from_string("{{ 'This is a string.' | with_prefix('Prefix: ', sep = ' : ') }}").render() 'Prefix : This is a string.'
with_suffix	metadock.env.MetadockEnv.with_suffix_filter: (self, value: str, suffix: str, sep: str = '') -> str	Filter which appends a suffix to a string, with an optional separator. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ 'This is a string' | with_suffix('Suffix') }}").render() 'This is a stringSuffix' >>> env.from_string("{{ 'This is a string' | with_suffix('Suffix', sep = ' : ') }}").render() 'This is a string : Suffix'
wrap	metadock.env.MetadockEnv.wrap_filter: (self, value: str, wrap: str) -> str	Filter which wraps an inner string with a given outer string. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> # Wrap with graves, like md.code(...) >>> env.from_string("{{ 'This is a string.' | wrap('`') }}").render() '`This is a string.`'
zip	metadock.env.MetadockEnv.zip_filter: (self, input_iterable: Iterable[Any], *iterables: Iterable[Any]) -> Iterable[tuple[Any, ...]]	Filter which zips an input iterable with one or more iterables. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ ['a', 'b', 'c'] | zip([1, 2, 3]) | list }}").render() "[('a', 1), ('b', 2), ('c', 3)]"

---

md namespace

Jinja Namespace for Markdown-related functions and filters.

Jinja macros

The following macros are available in the md namespace:

• md.blockquote
• md.code
• md.codeblock
• md.list
• md.tablehead
• md.tablerow

Jinja macro reference

Macro	Signature	Doc
md.blockquote	metadock.env.MetadockMdNamespace.blockquote: (self, content: str) -> str	Produces a Markdown blockquote from the given content by prepending each line with a gt symbol ("> "). >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ md.blockquote('This is a blockquote.') }}").render() '> This is a blockquote.'
md.code	metadock.env.MetadockMdNamespace.code: (self, content: str) -> str	Produces a Markdown inline code block from the given content by wrapping the string in graves ("`"). >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ md.code('This is an inline code block.') }}").render() 'This is an inline code block.'
md.codeblock	metadock.env.MetadockMdNamespace.codeblock: (self, content: str, language: str = '') -> str	Produces a Markdown codeblock from the given content by wrapping the string in triple-graves ("```"), and optionally specifies a language. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ md.codeblock('This is a codeblock.', language = 'sh') }}").render() 'sh\nThis is a codeblock.\n'
md.list	metadock.env.MetadockMdNamespace.list: (self, *items: str) -> str	Produces a Markdown list from the given content by prepending each line with a dash ("- "). If any of its arguments are, themselves, formatted as Markdown lists, then they are simply indented as sublists. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string( ... "{{ md.list('This is a list.', md.list('This is a sublist,', 'in two pieces.')) }}" ... ).render() '- This is a list.\n - This is a sublist,\n - in two pieces.'
md.tablehead	metadock.env.MetadockMdNamespace.tablehead: (self, *header_cells: str, bold: bool = False) -> str	Produces a Markdown table header from the given cells by joining each cell with pipes ("|") and wrapping the result in pipes, plus adding a header divider row. Cell contents have their pipes escaped with a backslash ("\"). To bold the header cell contents, supply bold = true. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string( ... "{{ md.tablehead('Column 1', 'Column 2', 'Column 3', bold = true) }}" ... ).render() '| <b>Column 1</b> | <b>Column 2</b> | <b>Column 3</b> |\n| --- | --- | --- |'
md.tablerow	metadock.env.MetadockMdNamespace.tablerow: (self, *row_cells: str) -> str	Produces a Markdown table row from the given cells by joining each cell with pipes ("|") and wrapping the result in pipes. Cell contents have their pipes escaped with a backslash ("\"). >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string( ... "{{ md.tablehead('Column 1', 'Column 2', 'Column 3') }}\n" ... "{{ md.tablerow('Value 1', 'Value 2', 'Value 3') }}" ... ).render() '| Column 1 | Column 2 | Column 3 |\n| --- | --- | --- |\n| Value 1 | Value 2 | Value 3 |'

Jinja filters

The following filters are available in the md namespace:

• md.convert
• md.list

Jinja filter reference

Filter	Signature	Doc
md.convert	metadock.env.MetadockMdNamespace.convert_filter: (self, md_content: str) -> str	Filter which converts Markdown content to HTML, by invoking marko.convert (using github-flavored md). >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ '# This is a heading\n\n> And a block quote.' | md.convert }}").render() ' This is a heading \n \n And a block quote. \n \n'
md.list	metadock.env.MetadockMdNamespace.list_filter: (self, values: str | Iterable[str]) -> str	Filter which unpacks an iterable of values into a Markdown list, or formats a single value as a Markdown list element. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string( ... "{{ ['This is a list.', 'This is a second element'] | md.list }}\n" ... ).render() '- This is a list.\n- This is a second element\n'

---

html namespace

Jinja namespace which owns HTML-related functions and filters.

Jinja macros

The following macros are available in the html namespace:

• html.bold
• html.code
• html.details
• html.italic
• html.pre
• html.summary
• html.underline

Jinja macro reference

Macro	Signature	Doc
html.bold	metadock.env.MetadockHtmlNamespace.bold: (self, content: str) -> str	Wraps a string in HTML bold tags (<b></b>). >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ html.bold('This is bold text.') }}").render() '<b>This is bold text.</b>'
html.code	metadock.env.MetadockHtmlNamespace.code: (self, content: str) -> str	Wraps a string in HTML code tags (<code></code>). >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ html.code('This is code text.') }}").render() '<code>This is code text.</code>'
html.details	metadock.env.MetadockHtmlNamespace.details: (self, *contents: str) -> str	Wraps a string in line-broken HTML details tags (<details></details>). Multiple arguments get separated by two line breaks. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ html.details('This is details text.') }}").render() '<details>\nThis is details text.\n</details>'
html.italic	metadock.env.MetadockHtmlNamespace.italic: (self, content: str) -> str	Wraps a string in HTML italic tags (<i></i>). >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ html.italic('This is italic text.') }}").render() '<i>This is italic text.</i>'
html.pre	metadock.env.MetadockHtmlNamespace.pre: (self, content: str, indent: int = 0) -> str	Wraps a string in preformatted HTML pre tags (<pre></pre>), and indents the content by the given amount. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ html.pre('This is code text.', indent = 4) }}").render() '<pre> This is code text.</pre>'
html.summary	metadock.env.MetadockHtmlNamespace.summary: (self, content: str) -> str	Wraps a string in line-broken HTML summary tags (<summary>\n\n</summary>). >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ html.summary('This is summary text.') }}").render() '<summary>\nThis is summary text.\n</summary>'
html.underline	metadock.env.MetadockHtmlNamespace.underline: (self, content: str) -> str	Wraps a string in HTML underline tags (<u></u>). >>> from metadock.env import MetadockEnv >>> env = MetadockEnv().jinja_environment() >>> env.from_string("{{ html.underline('This is underlined text.') }}").render() '<u>This is underlined text.</u>'

Jinja filters

The following filters are available in the html namespace:

• html.escape
• html.inline
• html.wrap_tag

Jinja filter reference

Filter	Signature	Doc
html.escape	metadock.env.MetadockHtmlNamespace.escape_filter: (self, content: str) -> str	Filter which escapes a string by replacing all HTML special characters with their HTML entity equivalents. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ '<p>This is a paragraph.</p>' | html.escape }}").render() '&lt;p&gt;This is a paragraph.&lt;/p&gt;'
html.inline	metadock.env.MetadockHtmlNamespace.inline_filter: (self, content: str) -> str	Filter which inlines a string by replacing all newlines with HTML line-breaks <br> singleton tags. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ 'This is a multi-line string.\nThis is the second line.\nAnd the third.' | html.inline }}").render() 'This is a multi-line string.<br>This is the second line.<br>And the third.'
html.wrap_tag	metadock.env.MetadockHtmlNamespace.wrap_tag_filter: (self, content: str, tag: str, attributes: dict = {}) -> str	Filter which wraps a string in a given HTML tag, with optional attributes. >>> from metadock.env import MetadockEnv >>> env = MetadockEnv(...).jinja_environment() >>> env.from_string("{{ 'This is a string.' | html.wrap_tag('p', {'id': 'my_paragraph') }}").render() '<p id="my_paragraph">This is a string.</p>'

---

Acknowledgements

Author:

• David Sillman [email protected]