From 6d82f48d2eeaef07f01b75d99b820751d0be4087 Mon Sep 17 00:00:00 2001 From: Sylvain Corlay Date: Thu, 27 Aug 2020 01:24:23 +0200 Subject: [PATCH] Add documentation for the nbconvert 6.0 template system --- docs/source/architecture.rst | 60 ++++-- docs/source/customizing.ipynb | 393 ---------------------------------- docs/source/customizing.rst | 128 +++++++++++ 3 files changed, 168 insertions(+), 413 deletions(-) delete mode 100644 docs/source/customizing.ipynb create mode 100644 docs/source/customizing.rst diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst index 8c5dc40cc..d8a4c7925 100644 --- a/docs/source/architecture.rst +++ b/docs/source/architecture.rst @@ -24,7 +24,7 @@ Format agnostic operations on cell content that do not violate the nbformat spec But often we want to have the notebook's structured content in a different format. Importantly, in many cases the structure of the notebook should be reflected in the structure of the output, adapted to the output's format. For that purpose, the original JSON structure of the document is crucial scaffolding needed to support this kind of structured output. -In order to maintain structured, it can be useful to apply our conversion programmatically on the structure itself. +In order to maintain structure, it can be useful to apply our conversion programmatically on the structure itself. To do so, when converting to formats other than the notebook, we use the `jinja`_ templating engine. The basic unit of structure in a notebook is the cell. @@ -96,16 +96,42 @@ Once a notebook is preprocessed, it's time to convert the notebook into the dest .. _templates_and_filters: -Templates and Filters ---------------------- +Templates +--------- + +Most Exporters in nbconvert are a subclass of `TemplateExporter`, which make use of +`jinja`_ to render a notebook into the destination format. + +Nbconvert templates can be selected from the command line with the ``--template`` +option. For example, to use the ``reveal`` template with the HTML exporter + +.. sourcecode:: bash + + jupyter nbconvert --to html --template reveal + +.. note:: + + Since version 6.0, The HTML exporter defaults to the ``lab`` template which produces + a DOM structure corresponding to the notebook component in JupyterLab. + + To produce HTML corresponding to the looks of the classic notebook, one can use the + ``classic`` template by passing ``--template classic`` to the command line. -Most Exporters in nbconvert are a subclass of `TemplateExporter`, -which means they use a `jinja`_ template to render a notebook into the destination format. -If you want to change how an exported notebook looks in an existing format, -a custom template is the place to start. +The nbconvet template system has been completely revamped with nbconvert 6.0 to allow +for greater extensibility. Nbconvert templates can now be installed as third-party packages +and are automatically picked up by nbconvert. -A jinja template is composed of blocks that look like this -(taken from nbconvert's default html template): +For more details about how to create custom templates, check out the :doc:`customizing` section +of the documentation. + +Filters +------- + +Filters are Python callables which take something (typically text) as an input, and produce a text output. +If you want to perform custom transformations of particular outputs, a filter may be the way to go. + +The following code snippet is an excert from the main default template of the HTML export. The displayed +block determines how text output on ``stdout`` is displayed in HTML. .. sourcecode:: html @@ -117,16 +143,10 @@ A jinja template is composed of blocks that look like this {%- endblock stream_stdout %} -This block determines how text output on ``stdout`` is displayed in HTML. -The ``{{- output.text | ansi2html -}}`` bit means -"Take the output text and pass it through ansi2html, then include the result here." -In this example, ``ansi2html`` is a `filter`_. -Filters are a jinja concept; they are Python callables which take something (typically text) as an input, and produce a text output. -If you want to perform new or more complex transformations of particular outputs, -a filter may be what you need. -Typically, filters are pure functions. -However, if you have a filter that itself requires some configuration, -it can be an instance of a callable, configurable class. +In the ``{{- output.text | ansi2html -}}`` bit, we invoke the ``ansi2html`` filter to transform the text output. + +Typically, filters are pure functions. However, filters that require some configuration, may be implemented as +Configurable classes. .. seealso:: @@ -135,6 +155,7 @@ it can be an instance of a callable, configurable class. Once it has passed through the template, an Exporter is done with the notebook, and returns the file data. + At this point, we have the file data as text or bytes and we can decide where it should end up. When you are using nbconvert as a library, as opposed to the command-line application, this is typically where you would stop, take your exported data, and go on your way. @@ -162,7 +183,6 @@ A *Postprocessor* is something that runs after everything is exported and writte The only postprocessor in nbconvert at this point is the `ServePostProcessor`, which is used for serving `reveal.js`_ HTML slideshows. - .. links: .. _jinja: http://jinja.pocoo.org/ diff --git a/docs/source/customizing.ipynb b/docs/source/customizing.ipynb deleted file mode 100644 index 426a49c63..000000000 --- a/docs/source/customizing.ipynb +++ /dev/null @@ -1,393 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Customizing nbconvert" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Under the hood, nbconvert uses [Jinja templates](http://jinja.pocoo.org/docs/latest/) to specify how the notebooks should be formatted. These templates can be fully customized, allowing you to use nbconvert to create notebooks in different formats with different styles as well." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Converting a notebook to an (I)Python script and printing to stdout\n", - "\n", - "Out of the box, nbconvert can be used to convert notebooks to plain Python files. For example, the following command converts the `example.ipynb` notebook to Python and prints out the result:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "!jupyter nbconvert --to python 'example.ipynb' --stdout" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "From the code, you can see that non-code cells are also exported. If you wanted to change that behaviour, you would first look to nbconvert [configuration options page](./config_options.rst) to see if there is an option available that can give you your desired behaviour. \n", - "\n", - "In this case, if you wanted to remove non-code cells from the output, you could use the `TemplateExporter.exclude_markdown` traitlet directly, as below. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "!jupyter nbconvert --to python 'example.ipynb' --stdout --TemplateExporter.exclude_markdown=True" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Custom Templates \n", - "\n", - "As mentioned above, if you want to change this behavior, you can use a custom template. The custom template inherits from the Python template and overwrites the markdown blocks so that they are empty. \n", - "\n", - "Below is an example of a custom template, which we write to a file called `simplepython.py.j2`. This template removes markdown cells from the output, and also changes how the execution count numbers are formatted:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%%writefile simplepython.py.j2\n", - "\n", - "{% extends 'python/index.py.j2'%}\n", - "\n", - "## remove markdown cells\n", - "{% block markdowncell %}\n", - "{% endblock markdowncell %}\n", - "\n", - "## change the appearance of execution count\n", - "{% block in_prompt %}\n", - "# [{{ cell.execution_count if cell.execution_count else ' ' }}]:\n", - "{% endblock in_prompt %}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Using this template, we see that the resulting Python code does not contain anything that was previously in a markdown cell, and only displays execution counts (i.e., `[#]:` not `In[#]:`):" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "!jupyter nbconvert --to python 'example.ipynb' --stdout --template-file=simplepython.py.j2" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Saving Custom Templates\n", - "\n", - "By default, nbconvert finds templates from a few locations.\n", - "\n", - "The recommended place to save custom templates, so that they are globally accessible to nbconvert, is your jupyter data directories:\n", - "\n", - "- share/jupyter\n", - " - nbconvert\n", - " - templates\n", - " - base\n", - " - classic\n", - " - lab\n", - " - latex\n", - " - markdown\n", - "\n", - "The HTML and LaTeX/PDF exporters will search the lab and latex subdirectories for templates by default, respectively.\n", - "\n", - "To find your jupyter configuration directory you can use:\n", - "\n", - "```python\n", - "from jupyter_core.paths import jupyter_path\n", - "print(jupyter_path('nbconvert','templates'))\n", - "```\n", - "\n", - "Additionally,\n", - "\n", - "```python\n", - "TemplateExporter.template_path=['.']\n", - "```\n", - "\n", - "defines an additional list of paths that nbconvert can look for user defined templates. It defaults to searching for custom templates in the current working directory and can be changed through configuration options." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Template structure\n", - "\n", - "Nbconvert templates consist of a set of nested blocks. When defining a new\n", - "template, you extend an existing template by overriding some of the blocks.\n", - "\n", - "All the templates shipped in nbconvert have the basic structure described here,\n", - "though some may define additional blocks." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from IPython.display import HTML\n", - "with open('template_structure.html') as f:\n", - " display(HTML(f.read()))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "See also [template_structure.html](template_structure.html)." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### A few gotchas\n", - "\n", - "Jinja uses `%`, `{`, and `}` for syntax by default which does not play nicely with LaTeX. In LaTeX, we have the following replacements:\n", - "\n", - "| Syntax | Default | LaTeX |\n", - "|----------|---------|---------|\n", - "| block | {% %} | ((* *)) |\n", - "| variable | {{ }} | ((( ))) |\n", - "| comment | {# #} | ((= =)) |" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "tags": [ - "Hard" - ] - }, - "source": [ - "## Templates using cell tags\n", - "\n", - "The notebook file format supports attaching arbitrary JSON metadata to each cell. In addition, every cell has a special `tags` metadata field that accepts a list of strings that indicate the cell's tags. To apply these, go to the `View → CellToolbar → Tags` option which will create a Tag editor at the top of every cell. \n", - "\n", - "First choose a notebook you want to convert to html, and apply the tags: `\"Easy\"`, `\"Medium\"`, or \n", - "`\"Hard\"`. \n", - "\n", - "With this in place, the notebook can be converted using a custom template.\n", - "\n", - "Design your template in the cells provided below.\n", - "\n", - "Hint: tags are located at `cell.metadata.tags`, the following Python code collects the value of the tag: \n", - "\n", - "```python\n", - "cell['metadata'].get('tags', [])\n", - "```\n", - "\n", - "Which you can then use inside a Jinja template as in the following:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%%writefile mytemplate.html.j2\n", - "\n", - "{% extends 'lab/index.html.j2'%}\n", - "{% block any_cell %}\n", - "{% if 'Hard' in cell['metadata'].get('tags', []) %}\n", - "
\n", - " {{ super() }}\n", - "
\n", - "{% elif 'Medium' in cell['metadata'].get('tags', []) %}\n", - "
\n", - " {{ super() }}\n", - "
\n", - "{% elif 'Easy' in cell['metadata'].get('tags', []) %}\n", - "
\n", - " {{ super() }}\n", - "
\n", - "{% else %}\n", - " {{ super() }}\n", - "{% endif %}\n", - "{% endblock any_cell %}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now, if we collect the result of using nbconvert with this template, and display the resulting html, we see the following:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "example = !jupyter nbconvert --to html 'example.ipynb' --template-file=mytemplate.html.j2 --stdout\n", - "example = example[3:] # have to remove the first three lines which are not proper html\n", - "from IPython.display import HTML\n", - "display(HTML('\\n'.join(example))) " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Templates using custom cell metadata \n", - "\n", - "We demonstrated [above](#Templates-using-cell-tags) how to use cell tags in a template to apply custom styling to a notebook. But remember, the notebook file format supports attaching _arbitrary_ JSON metadata to each cell, not only cell tags. \n", - "Here, we describe an exercise for using an `example.difficulty` metadata field (rather than cell tags) to do the same as before (to mark up different cells as being \"Easy\", \"Medium\" or \"Hard\").\n", - "\n", - "### How to edit cell metadata\n", - "\n", - "To edit the cell metadata from within the notebook, go to the menu item: `View → Cell Toolbar → Edit Metadata`. This will bring up a toolbar above each cell with a button that says \"Edit Metadata\". Click this button, and a field will pop up in which you will directly edit the cell metadata JSON. \n", - "\n", - "**NB**: Because it is JSON, you will need to ensure that what you write is valid JSON. \n", - "\n", - "### Template challenges: dealing with missing custom metadata fields\n", - "\n", - "One of the challenges of dealing with custom metadata is to handle the case where the metadata is not present on every cell. This can get somewhat tricky because of JSON objects tendency to be deeply nested coupled with Python's (and therefore Jinja's) approach to calling into dictionaries. Specifically, the following code will error:\n", - "\n", - "```python\n", - "foo = {}\n", - "foo[\"bar\"]\n", - "```\n", - "\n", - "Accordingly, it is better to use the [{}.get()](https://docs.python.org/3.6/library/stdtypes.html#dict.get) method which allows you to set a default value to return if no key is found as the second argument. \n", - "\n", - "Hint: if your metadata items are located at `cell.metadata.example.difficulty`, the following Python code would get the value defaulting to an empty string (`''`) if nothing is found:\n", - "\n", - "```python\n", - "cell['metadata'].get('example', {}).get('difficulty', '')\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise: Write a template for handling custom metadata\n", - "Now, write a template that will look for `Easy`, `Medium` and `Hard` metadata values for the `cell.metadata.example.difficulty` field and wrap them in a div with a green, orange, or red thin solid border (respectively). \n", - "\n", - "**NB**: This is the same design and logic as used in the previous cell tag example.\n", - "\n", - "#### How to get `example.ipynb`\n", - "\n", - "We have provided an example file in `example.ipynb` in the nbconvert documentation that has already been marked up with both tags and the above metadata for you to test with. You can get it from [this link to the raw file](example.ipynb) or by cloning the repository [from GitHub](https://github.com/jupyter/nbconvert) and navingating to `nbconvert/docs/source/example.ipynb`. \n", - "\n", - "#### Convert `example.ipynb` using cell tags \n", - "\n", - "First, make sure that you can reproduce the previous result using the cell tags template that we have provided above. \n", - "\n", - "**Easy**: If you want to make it easy on yourself, create a new file `my_template.html.j2` in the same directory as `example.ipynb` and copy the contents of the cell we use to write `mytemplate.html.j2` to the file system. \n", - "\n", - "Then run `jupyter nbconvert --to html 'example.ipynb' --template-file=mytemplate.html.j2` and see if your template file works. \n", - "\n", - "\n", - "**Moderate**: If you want more of a challenge, try recreating the jinja template by modifying the following jinja template file:\n", - "\n", - "```python\n", - "{% extends 'lab/base.html.j2'%}\n", - "{% block any_cell %}\n", - "
\n", - " {{ super() }}\n", - "
\n", - "{% endblock any_cell %}\n", - "```\n", - "\n", - "**Moderate**: If you want custom config or a composition of files, you'll want to instead make a template directory. Create the directory `mytemplate` and add the following files to it:\n", - "\n", - "- `index.html.j2` -- copy the contents of the cell we use to write `mytemplate.html.j2` to the file system.\n", - "- `conf.json` -- add the following contents to set the default configuration options\n", - "```json\n", - "{\n", - " \"base_template\": \"lab\",\n", - " \"mimetypes\": {\n", - " \"text/html\": true\n", - " }\n", - "}\n", - "```\n", - "\n", - "Then run `jupyter nbconvert --to html 'example.ipynb' --template=mytemplate` and see if your package works. This option allows for much more flexibility and combination of configuration and additional config files to support more advanced templates.\n", - "\n", - "\n", - "**Hard**: If you want even more of a challenge, try recreating the jinja template from scratch. \n", - "\n", - "#### Write your template\n", - "\n", - "Once you've done at least the **Easy** version of the previous step, try modifying your template to use `cell.metadata.example.difficulty` fields rather than cell tags. \n", - "\n", - "#### Convert `example.ipynb` with formatting from custom metadata\n", - "\n", - "Once you've written your template, try converting `example.ipynb` using the following command (making sure that `your_template.html.j2` is in your local directory where you are running the command):\n", - "\n", - "```bash\n", - "jupyter nbconvert --to html 'example.ipynb' --template=your_template.html.j2 --stdout\n", - "```\n", - "\n", - "The resulting display should pick out different cells to be bordered with green, orange, or red.\n", - "\n", - "If you do that successfullly, the resulting html document should look like the following\n", - "(see also [example-custom-metadata.html](example-custom-metadata.html)): " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from IPython.display import IFrame\n", - "\n", - "IFrame('example-custom-metadata.html', width='100%', height=600)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.5" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/docs/source/customizing.rst b/docs/source/customizing.rst new file mode 100644 index 000000000..7a2d99ef7 --- /dev/null +++ b/docs/source/customizing.rst @@ -0,0 +1,128 @@ +Creating Custom Templates for nbconvert +======================================= + +Selecting a template +-------------------- + +Most Exporters in nbconvert are a subclass of `TemplateExporter`, which make use of +`jinja`_ to render a notebook into the destination format. + +Nbconvert templates can be selected from the command line with the ``--template`` +option. For example, to use the ``reveal`` template with the HTML exporter + +.. sourcecode:: bash + + jupyter nbconvert --to html --template reveal + +Where are nbconvert templates installed? +---------------------------------------- + +Nbconvert templates are *directories* containing resources for nbconvert template +exporters such as inja templates, and associated assets. They are installed in the +**data directory** of nbconvert, namely ``/jupyter/nbconvert``. +Nbconvert includes several templates already. + +For example, three templates are provided for the HTML exporter: + + - ``lab`` (The default HTML template, which produces the same DOM structure as JupyterLab) + - ``classic`` (The HTML template styled after the classic notebook) + - ``reveal`` (For producing slideshows). + +.. note:: + + Running ``jupyter --paths`` will show all Jupyter directories and search paths. + + For example, on Linux, ``jupyter --paths`` returns: + + .. code:: + + $ jupyter --paths + config: + /home//.jupyter + //etc/jupyter + /usr/local/etc/jupyter + /etc/jupyter + data: + /home//.local/share/jupyter + //share/jupyter + /usr/local/share/jupyter + /usr/share/jupyter + runtime: + /home//.local/share/jupyter/runtime + + +The content of nbconvert templates +---------------------------------- + +conf.json +~~~~~~~~~ + +Nbconvert templates all include a ``conf.json`` file at the root of the directory, +which is used to indicate + + - the base template that we are inheriting from. + - the mimetypes produced by the template. + - preprocessors classes to register in the exporter when using that template. + +Inspecting the configuration of the reveal template we see that it inherits from the lab +template, exports text/html, and enables a preprocessor called "500-reveal". + +.. code:: + + { + "base_template": "lab", + "mimetypes": { + "text/html": true + }, + "preprocessors": { + "500-reveal": { + "type": "nbconvert.exporters.slides._RevealMetadataPreprocessor", + "enabled": true + } + } + } + +Inheitance +~~~~~~~~~~~~ + +Nbconvert walks up the inheritance structure determined by ``conf.json`` and produces an agregated +configuration, merging the dictionaries of registered preprocessors. +The lexical ordering of the preprocessors by name determines the order in which they will be run. + +Besides the ``conf.json`` file, nbconvert templates most typically include jinja templates files, +although any other resource from the base template can be overriden in the derived template. + +For example, inspecting the content of the ``classic`` template located in +``share/jupyter/nbconvert/templates/classic``, we find the following content: + +Inspecting the content of the ``classic`` template we find the following file structure: + +.. code:: + + share/jupyter/nbconvert/templates/classic + ├── static + │ └── styles.css + ├── conf.json + ├── index.html.j2 + └── base.html.j2 + +The ``classic`` template exporter includes a ``index.html.j2`` jinja template (which is the main entry point +for HTML exporters) as well as CSS and a base template file in ``base.html.j2``. + +.. note:: + + A template inheriting from ``classic`` would specify ``"base_template": "classic"`` and could + override any of these files. For example, one could make a "classiker" template merely providing + an alternative ``styles.css`` file. + +Inheritance in Jinja +~~~~~~~~~~~~~~~~~~~~ + +In nbconvert, jinja templates can inherrit from any other jinja template available in its current directory +or base template directory by name. Jinja templates of other directories can be addressed by their relative path +from the Jupyter data directory. + +For example, in the reveal template, ``index.html.j2`` extends ``base.html.j2`` which is in the same directory, and +``base.html.j2`` extends ``lab/base.html.j2``. This approach allows using content that is available in other templates +or may be overriden in the current template. +