Fix some doc warnings and convert to myst (#482)

* Fix some doc warnings and convert to myst

* Add orphan to xgboost notebook

* Add ipywidgets as dependency for examples

* Updated doc/README.md and corrected display math

* Update to sphinx 4.x and revert numerous math changes

* Corrected TreeSHAP eqn tag

* Change admonitions to notes. Fix bullets. Tidy requirements/docs.txt.

Makefile

@@ -17,13 +17,13 @@ mypy:
 .PHONY: build_docs
 build_docs:
 	# readthedocs.org build command
-	python -m sphinx -T -b html -d _build/doctrees -D language=en doc/source  doc/_build/html
+	python -m sphinx -T -b html -d doc/_build/doctrees -D language=en doc/source  doc/_build/html
 
 .PHONY: build_latex
 build_latex: ## Build the documentation into a pdf
 	# readthedocs.org build command
 	# explicit cd here due to a bug in latexmk 4.41
-	python -m sphinx -b latex -d _build/doctrees -D language=en doc/source doc/_build/latex && \
+	python -m sphinx -b latex -d doc/_build/doctrees -D language=en doc/source doc/_build/latex && \
 	cd doc/_build/latex && \
 	latexmk -pdf -f -dvi- -ps- -jobname=alibi -interaction=nonstopmode
 

alibi/explainers/anchor_text.py

@@ -1203,6 +1203,7 @@ def __init__(self,
         kwargs
             Sampling parameters can be passed as `kwargs` depending on the `sampling_strategy`.
             Check default parameters defined in:
+
                 - :py:data:`alibi.explainers.anchor_text.DEFAULT_SAMPLING_UNKNOWN`
                 - :py:data:`alibi.explainers.anchor_text.DEFAULT_SAMPLING_SIMILARITY`
                 - :py:data:`alibi.explainers.anchor_text.DEFAULT_SAMPLING_LANGUAGE_MODEL`

alibi/utils/lang_model.py

@@ -119,7 +119,7 @@ def select_word(self,
         """
         Given a tokenized text and the starting index of a word, the function selects the entire word.
         Note that a word is composed of multiple tokens (e.g., word = [head_token tail_token_1
-        tail_token_2 ... tail_token_ k]. The tail tokens can be identified based on the
+        tail_token_2 ... tail_token_k]. The tail tokens can be identified based on the
         presence/absence of SUBWORD_PREFIX. See `is_subword_prefix` for more details.
 
         Parameters

doc/README.md

@@ -1,7 +1,7 @@
 # Documentation for alibi
 
-This directory contains the sources (`.md`, `.rst` and `.ipynb` files) for the
-documentation. The main index page is defined in `source/index.rst`.
+This directory contains the sources (`.md` and `.ipynb` files) for the
+documentation. The main index page is defined in `source/index.md`.
 The Sphinx options and plugins are found in the `source/conf.py` file.
 The documentation is generated in full by calling `make html` which
 also automatically generates the Python API documentation from
@@ -25,11 +25,17 @@ We are now ready to build the docs:
 
 This calls the sphinx html builder command defined in the main repo [Makefile](../Makefile). Note this can take some time as some of the notebooks may be executed
 during the build process. The resulting documentation is located in the
-`_build` directory with `_build/html/index.html` marking the homepage.
+`_build` directory with `_build/html/index.html` marking the homepage. 
+
+A pdf of the docs can also be built:
+
+`make -C .. build_latex`
+
+with the resulting pdf file residing at `_build/latex/alibi.pdf`.
 
 ## Sphinx extensions and plugins
-We use various Sphinx extensions and plugins to build the documentation:
- * [m2r](https://github.com/miyakogi/m2r) - to handle both `.rst` and `.md`
+We use various Sphinx extensions and plugins to build the documentation, including:
+ * [myst-parser](https://myst-parser.readthedocs.io/en/latest/) - to allow `rst` directives in Markdown files (see [here](#myst-format))
  * [sphinx.ext.napoleon](https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html) - support extracting Numpy style doctrings for API doc generation
  * [sphinx_autodoc_typehints](https://github.com/agronholm/sphinx-autodoc-typehints) - support parsing of typehints for API doc generation
  * [sphinxcontrib.apidoc](https://github.com/sphinx-contrib/apidoc) - automatic running of [sphinx-apidoc](https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html) during the build to document API
@@ -47,10 +53,36 @@ All examples are Jupyter notebooks and live in the top level `examples` director
 ```
 From here on you can link and refer to the `notebook.ipynb` elsewhere in the documentation as if it lived under `doc/source/examples`.
 
+## MyST format
+
+For consistency, all docs files excluding `.ipynb` notebooks are written as Markedly Structured Text (MyST) files. 
+Although still named as `.md` files, the files are passed through [myst-parser](https://myst-parser.readthedocs.io/en/latest/), 
+which processes `rst`-type directives contained within the `.md` files. This allows for more powerful functionality 
+to be included in the docs. For example, an admonition block can be included in a `.md` file with:
+
+````md
+```{admonition} My markdown link
+Here is [markdown link syntax](https://jupyter.org)
+```
+````
+
+and an image can be embedded with:
+
+````md
+```{image} img/fun-fish.png
+:alt: fishy
+:class: bg-primary
+:width: 200px
+:align: center
+```
+````
+
+For more details on the MyST syntax see the [MyST docs](https://myst-parser.readthedocs.io/en/latest/syntax/syntax.html). 
+
 ## Gotchas when writing notebooks as examples
 We use Jupyter notebooks for examples and method descriptions and invoke the [nbsphinx](https://nbsphinx.readthedocs.io) plugin, which in turn invokes `pandoc` for rendering the notebooks as static documentation. Generally, the Jupyter notebook is more permissive for what it can render correctly than the static documentation, so it is important to check that the content is rendered correctly in the static docs as well. Here is a list of common formatting gotchas and how to fix them:
 * When using a bullet-point list, leave a blank line before the preceding paragraph, otherwise it will fail to render
-* Always use `$$ $$` or `\begin{equation}\end{equation}`to delimit display math
+* Always use `$$ $$` or `\begin{equation}\end{equation}`to delimit display math.
 * Leave a blank line before and after any display math
 * For references and footnotes, the tag indicating the section needs to start with an uppercase letter, e.g. `[[1]](#References)` linking to a section `<a id='References'></a>
 [1](#f_1) reference here`

doc/source/conf.py

@@ -49,13 +49,12 @@
     'sphinx.ext.mathjax',
     'sphinx.ext.ifconfig',
     'sphinx.ext.viewcode',
-    #'recommonmark',
     'sphinx.ext.napoleon',
     'sphinx_autodoc_typehints',
     'sphinxcontrib.apidoc',  # automatically generate API docs, see https://github.com/rtfd/readthedocs.org/issues/1139
     'nbsphinx',
     'nbsphinx_link',  # for linking notebooks from outside sphinx source root
-    'm2r'
+    'myst_parser'
 ]
 
 # nbsphinx settings
@@ -137,11 +136,7 @@
 html_static_path = ['_static']
 
 # override default theme width
-html_context = {
-    'css_files': [
-        '_static/theme_overrides.css',  # override wide tables in RTD theme
-    ],
-}
+html_css_files = ["theme_overrides.css"]
 
 # Custom sidebar templates, must be a dictionary that maps document names
 # to template names.
@@ -281,3 +276,18 @@
         """ +
 git_rev + r"{{ docpath }}"
 )
+
+# -- myst-parser configuration -----------------------------------------------
+# See https://myst-parser.readthedocs.io/en/latest/syntax/optional.html for 
+# details of available extensions.
+myst_enable_extensions = [
+    "dollarmath",
+    "amsmath",
+    "colon_fence",
+    "smartquotes",
+    "tasklist",
+    "html_image",
+]
+# Create heading anchors for h1 to h3 (useful for local toc's)
+myst_heading_anchors = 3
+

doc/source/index.md

@@ -0,0 +1,85 @@
+% alibi documentation master file, created by
+% sphinx-quickstart on Thu Feb 28 11:04:41 2019.
+% You can adapt this file completely to your liking, but it should at least
+% contain the root `toctree` directive.
+
+```{include} landing.md
+```
+
+```{toctree}
+:caption: Overview
+:maxdepth: 1
+
+overview/getting_started
+overview/algorithms
+overview/white_box_black_box
+overview/saving
+overview/roadmap
+```
+
+```{toctree}
+:caption: Methods
+:maxdepth: 1
+
+methods/ALE.ipynb
+methods/Anchors.ipynb
+methods/CEM.ipynb
+methods/CF.ipynb
+methods/CFProto.ipynb
+methods/CFRL.ipynb
+methods/IntegratedGradients.ipynb
+methods/KernelSHAP.ipynb
+methods/LinearityMeasure.ipynb
+methods/TrustScores.ipynb
+methods/TreeSHAP.ipynb
+```
+
+```{toctree}
+:caption: Examples
+:maxdepth: 1
+
+examples/ale_regression_boston
+examples/ale_classification
+examples/anchor_tabular_adult
+examples/anchor_tabular_iris
+examples/anchor_text_movie
+examples/anchor_image_imagenet
+examples/anchor_image_fashion_mnist
+examples/cem_mnist
+examples/cem_iris
+examples/cf_mnist.ipynb
+examples/cfproto_mnist.ipynb
+examples/cfproto_housing.ipynb
+examples/cfproto_cat_adult_ohe.ipynb
+examples/cfproto_cat_adult_ord.ipynb
+examples/cfrl_mnist.ipynb
+examples/cfrl_adult.ipynb
+examples/kernel_shap_wine_intro
+examples/kernel_shap_wine_lr
+examples/kernel_shap_adult_lr
+examples/kernel_shap_adult_categorical_preproc
+examples/distributed_kernel_shap_adult_lr
+examples/linearity_measure_iris
+examples/linearity_measure_fashion_mnist
+examples/trustscore_iris
+examples/trustscore_mnist
+examples/interventional_tree_shap_adult_xgb
+examples/path_dependent_tree_shap_adult_xgb
+examples/integrated_gradients_imagenet.ipynb
+examples/integrated_gradients_mnist.ipynb
+examples/integrated_gradients_imdb.ipynb
+examples/integrated_gradients_transformers.ipynb
+```
+
+```{toctree}
+:caption: API reference
+:maxdepth: 1
+
+API reference <api/modules>
+```
+
+# Indices and tables
+
+- {ref}`genindex`
+- {ref}`modindex`
+- {ref}`search`

doc/source/landing.md

@@ -1,6 +1,12 @@
-<p align="center">
-  <img src="_static/Alibi_Explain_Logo_rgb.png" alt="Alibi Logo" width="50%">
-</p>
+---
+orphan: true
+---
+
+```{image} _static/Alibi_Explain_Logo_rgb.png
+:align: center
+:alt: Alibi Logo
+:width: 50%
+```
 
 [Alibi](https://github.com/SeldonIO/alibi) is an open source Python library aimed at machine learning model inspection and interpretation.
 The focus of the library is to provide high-quality implementations of black-box, white-box, local and global

doc/source/methods/CEM.ipynb

@@ -272,7 +272,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "\\begin{equation*} \\frac{\\partial L}{\\partial x} = \\frac{\\partial L}{\\partial p} \\frac{\\partial p}{\\partial x} \\end{equation*}\n"
+    "$$\n",
+    "\\frac{\\partial L}{\\partial x} = \\frac{\\partial L}{\\partial p} \\frac{\\partial p}{\\partial x}\n",
+    "$$"
    ]
   },
   {
@@ -311,7 +313,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -325,7 +327,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.5"
+   "version": "3.8.11"
   },
   "varInspector": {
    "cols": {

doc/source/methods/CF.ipynb

@@ -221,7 +221,8 @@
     "\n",
     "In this case, we need to evaluate the gradients of the loss function with respect to the input features $X$ numerically:\n",
     "    \n",
-    "\\begin{equation*} \\frac{\\partial L_{\\text{pred}}}{\\partial X} = \\frac{\\partial L_\\text{pred}}{\\partial p} \\frac{\\partial p}{\\partial X} \\end{equation*}\n",
+    "$$\\frac{\\partial L_{\\text{pred}}}{\\partial X} = \\frac{\\partial L_\\text{pred}}{\\partial p} \\frac{\\partial p}{\\partial X}\n",
+    "$$\n",
     "\n",
     "where $L_\\text{pred}$ is the predict function loss term, $p$ the predict function and $x$ the input features to optimize. There is now an additional hyperparameter to consider:\n",
     "\n",
@@ -249,7 +250,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -263,7 +264,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.5"
+   "version": "3.8.11"
   },
   "varInspector": {
    "cols": {