Documentation deployment

Documentation for all MDAnalysis repositories should be self-deployed if served from the mdanalysis.org URL, instead of using ReadTheDocs (RTD). While RTD is simple and easy to use, it is not compliant with our Privacy Policy, as it adds Google Analytics and cookies (https://github.com/MDAnalysis/mdacli/issues/77). Google Analytics can be toggled off, but it's less apparent what to do about cookies.

Therefore we have rolled our own deployment, adapted from MDTraj's approach. After doing the GitHub and DNS setup and setting up conf.py, this comprises the following steps:

1. Build documentation (typically done via a CI approach; formerly Travis but now Github Actions)
2. Move documentation into a $version subfolder; i.e. cd package/doc/html/html; mkdir 2.0.1; mv *.* 2.0.1
3. Update versions.json and write or update the sitemap index
4. Push documentation directory to the gh-pages branch

Below is an explanation of the different parts of the documentation, followed by a more detailed list of steps and where they happen.

GitHub settings and CNAME records

The set-up should be that the repository serves a GitHub Pages site from the gh-pages branch. The custom domain should be set in GitHub Settings > Pages. You can use either a subdomain (e.g. docs.mdanalysis.org/) or a subdirectory (e.g. mdanalysis.org/docs). If you choose a subdirectory, no further work is required; a CNAME record for www.mdanalysis.org is already set up to point to mdanalysis.github.io. If you choose a subdomain, you need to add a new CNAME record in Cloudflare > DNS. The record should be for the subdomain part only, and point to mdanalysis.github.io. e.g., the documentation CNAME record is from docs -> mdanalysis.github.io . What this means is that the contents of the gh-pages branch is available at docs.mdanalysis.org.

A selection of core devs should have access to the Cloudflare account, which is under Oliver's ASU email address.

See https://github.com/MDAnalysis/MDAnalysis.github.io/wiki/DNS for more and please update it if more records are added.

Required extensions

These extensions need to be installed to build the sitemap and update the theme.

• sphinx-sitemap
• msmb_theme
• sphinx_rtd_theme

conf.py settings

These settings need to be in conf.py:

import msmb_theme

extensions = [..., 'sphinx_sitemap', 'sphinx_rtd_theme']

site_url = "https://docs.mdanalysis.org/"

html_theme = 'msmb_theme'

html_theme_path = [
    msmb_theme.get_html_theme_path(),
    sphinx_rtd_theme.get_html_theme_path()
]

html_context = {
    'versions_json_url': os.path.join(site_url, "versions.json")  # this should be the default I think
}

Documentation parts

This section contains a description of different parts of the documentation, in no particular order.

Actual documentation

The actual documentation is what gets built when you run make html. These comprise the actual HTML files that you can open in your browser. They are usually built in the root documentation directory: package/doc/html/html for the MDAnalysis/mdanalysis repo.

Sitemaps

Sitemaps are XML files that are used to provide information about the pages of the documentation. These are for search engines to see what pages have to offer and do a better job of providing results. Sitemaps are built as part of the documentation, because we use the sphinx-sitemap extension.

Sitemaps can only contain URLs within the directory. e.g. a sitemap for the 1.0.1 version cannot contain links to the 2.0.0 version. Therefore, we also need a sitemap index file. This is another XML file that contains links to the individual version sitemaps. The sitemap index file is written by us, in update_json_stubs_sitemap.py.

versions.json

Because we are after versioned documentation, we install msmb_theme for the little versions popup to navigate between the documentation. It is an add-on to sphinx_rtd_theme, which must also be installed. The docs use versions.json to show the individual versions. The data in versions.json is a list of these dictionaries:

{"version": "0.20.1",
 "display": "0.20.1",  # this is how the version is labelled in the popup
 "url": "https://docs.mdanalysis.org/0.20.1",  # link to documentation for it
 "latest": False  # whether it's the latest version or not}

versions.json should be in the root directory of the documentation. i.e., for docs.mdanalysis.org, it's at https://docs.mdanalysis.org/versions.json -- you can actually click that link and view the file.

We write and edit versions.json in update_json_stubs_sitemap.py.

stable and dev versions

It's convenient to maintain up-to-date documentation at a constant URL for links and intersphinx. For MDAnalysis/mdanalysis, this is done at https://docs.mdanalysis.org/stable/ and https://docs.mdanalysis.org/dev/ . Just like the 1.0.1 docs are in a 1.0.1 folder, the stable and dev are just folders of documentation. These are copies of whichever version has been identified to be the latest release, and latest development version. Copying these directories is done in update_json_stubs_sitemap.py.

HTML Redirects

An HTML redirect is a short page that immediately goes to another page. It looks like this:

<!DOCTYPE html>
<meta charset="utf-8">
<title>Redirecting to {url}</title>
<meta http-equiv="refresh" content="0; URL={url}">
<link rel="canonical" href="{url}">

We use this to redirect pages such as docs.mdanalysis.org to docs.mdanalysis.org/stable. This is likely all that new MDAnalysis repositories will need. However, for repositories that use to serve documentation without versions, we can also write redirect stubs for all the stable documentation.

e.g. the original MDAnalysis documentation did not have versions, so links to documentation looked like this: https://docs.mdanalysis.org/documentation_pages/topology_modules

For backwards compatibility, update_json_stubs_sitemap.py recursively goes through all the documentation in stable and writes redirect stubs for all of them. An example for the above page is https://github.com/MDAnalysis/mdanalysis/blob/gh-pages/documentation_pages/topology_modules.html . If you click on https://docs.mdanalysis.org/documentation_pages/topology_modules , it should redirect to https://docs.mdanalysis.org/stable/documentation_pages/topology_modules .

We also redirect from /latest/index.html to the actual latest version, also for legacy reasons.

CI and scripts

Most of the documentation parts are added or modified in the CI or in update_json_stubs_sitemap.py.

These are the following steps:

• CI: build documentation (usually done as a test of whether documentation builds)
• CI: determine version number. This can be done by importing the package if it's installed, or string parsing something. Set it to $VERSION
• CI: Go into the built documentation directory. From now on this is the root directory for gh-pages. All version directories are called subdirectories.
• CI: make a directory called $VERSION and move everything into $VERSION (mkdir ../${VERSION} && mv * ../${VERSION} && mv ../${VERSION} $VERSION
• CI: Fetch the gh-pages branch into the root directory.
• CI: Call update_json_stubs_sitemap.py
  • Python: Download versions.json and add new version to it
  • Python: Copy version subdirectory to stable/ or dev/ if the version qualifies as the latest version or a development version
  • Python: Create a sitemap index of individual sitemaps.
• CI: touch all files and create a file called .nojekyll to tell GitHub not to do Jekyll processing
• CI: git add -A the version subdirectory -- this means to add changes from all tracked and untracked files. git add all other relevant files, e.g. .nojekyll versions.json *.xml *.html index.html.
• CI: if any of dev, stable, documentation_pages exist, add those too. The last one is probably MDAnalysis/mdanalysis only.
• CI: commit and push to gh-pages. The current CI creates a dummy committer called github-actions.