Merge pull request #9150 from readthedocs/humitos/build-commands

docs/user/build-customization.rst

@@ -3,7 +3,18 @@ Build customization
 
 Read the Docs has a :doc:`well-defined build process <builds>` that works for many projects,
 but we offer additional customization to support more uses of our platform.
-This page explains how to extend the build process, using :term:`user-defined build jobs` to execute custom commands.
+This page explains how to extend the build process using :term:`user-defined build jobs` to execute custom commands,
+and also how to override the build process completely:
+
+`Extend the build process`_
+    If you are using Sphinx or Mkdocs and need to execute additional commands.
+
+`Override the build process`_
+    If you want full control over your build. This option supports any tool that generates HTML as part of the build.
+
+
+Extend the build process
+------------------------
 
 In the normal build process,
 the pre-defined jobs ``checkout``, ``system_dependencies``, ``create_environment``, ``install``, ``build`` and ``upload`` are executed.
@@ -37,7 +48,8 @@ These jobs can be declared by using a :doc:`/config-file/index` with the :ref:`c
 Let's say the project requires commands to be executed *before* installing any dependency into the Python environment and *after* the build has finished.
 In that case, a config file similar to this one can be used:
 
-.. code:: yaml
+.. code-block:: yaml
+   :caption: .readthedocs.yaml
 
    version: 2
    build:
@@ -63,7 +75,7 @@ There are some caveats to knowing when using user-defined jobs:
 
 
 Examples
---------
+++++++++
 
 We've included some common examples where using :ref:`config-file/v2:build.jobs` will be useful.
 These examples may require some adaptation for each projects' use case,
@@ -77,7 +89,8 @@ Read the Docs does not perform a full clone on ``checkout`` job to reduce networ
 Because of this, extensions that depend on the full Git history will fail.
 To avoid this, it's possible to unshallow the clone done by Read the Docs:
 
-.. code:: yaml
+.. code-block:: yaml
+   :caption: .readthedocs.yaml
 
    version: 2
    build:
@@ -94,7 +107,8 @@ Generate documentation from annotated sources with Doxygen
 
 It's possible to run Doxygen as part of the build process to generate documentation from annotated sources:
 
-.. code:: yaml
+.. code-block:: yaml
+   :caption: .readthedocs.yaml
 
    version: 2
    build:
@@ -114,7 +128,8 @@ Use MkDocs extensions with extra required steps
 There are some MkDocs extensions that require specific commands to be run to generate extra pages before performing the build.
 For example, `pydoc-markdown <http://niklasrosenstein.github.io/pydoc-markdown/>`_
 
-.. code:: yaml
+.. code-block:: yaml
+   :caption: .readthedocs.yaml
 
    version: 2
    build:
@@ -135,7 +150,8 @@ In case this happens and the project is using any kind of extension that generat
 this could cause an invalid version number to be generated.
 In that case, the Git index can be updated to ignore the files that Read the Docs has modified.
 
-.. code:: yaml
+.. code-block:: yaml
+   :caption: .readthedocs.yaml
 
    version: 2
    build:
@@ -154,7 +170,8 @@ Sphinx comes with a `linkcheck <https://www.sphinx-doc.org/en/master/usage/build
 This helps ensure that all external links are still valid and readers aren't linked to non-existent pages.
 
 
-.. code:: yaml
+.. code-block:: yaml
+   :caption: .readthedocs.yaml
 
    version: 2
    build:
@@ -173,7 +190,8 @@ In case the repository contains large files that are tracked with Git LFS,
 there are some extra steps required to be able to download their content.
 It's possible to use ``post_checkout`` user-defined job for this.
 
-.. code:: yaml
+.. code-block:: yaml
+   :caption: .readthedocs.yaml
 
    version: 2
    build:
@@ -204,7 +222,8 @@ Install Node.js dependencies
 It's possible to install Node.js together with the required dependencies by using :term:`user-defined build jobs`.
 To setup it, you need to define the version of Node.js to use and install the dependencies by using ``build.jobs.post_install``:
 
-.. code:: yaml
+.. code-block:: yaml
+   :caption: .readthedocs.yaml
 
    version: 2
    build:
@@ -218,3 +237,39 @@ To setup it, you need to define the version of Node.js to use and install the de
          - npm ci
          # Install any other extra dependencies to build the docs
          - npm install -g jsdoc
+
+
+Override the build process
+--------------------------
+
+.. warning::
+
+   This feature is in a *beta phase* and could suffer incompatible changes or even removed completely in the near feature.
+   It does not yet support some of the Read the Docs' features like the :term:`flyout menu`, search and ads.
+   However, we do plan to support these features in the future.
+   Use this feature at your own risk.
+
+
+If your project requires full control of the build process,
+and :ref:`extending the build process <build-customization:extend the build process>` is not enough,
+all the commands executed during builds can be overridden using the :ref:`config-file/v2:build.commands` configuration file key.
+
+For example, if your project uses `Pelican <https://blog.getpelican.com/>`_ instead of Sphinx for its documentation,
+your project could use the following configuration file:
+
+.. code-block:: yaml
+   :caption: .readthedocs.yaml
+
+   version: 2
+   build:
+     os: "ubuntu-22.04"
+     tools:
+       python: "3.10"
+     commands:
+       - pip install pelican[markdown]
+       - pelican --settings docs/pelicanconf.py --output _readthedocs/html/ docs/
+
+
+As Read the Docs does not have control over the build process,
+you are responsible for running all the commands required to install requirements and build the documentation properly.
+Once the build process finishes, the ``_readthedocs/html/`` folder will be hosted.

docs/user/config-file/v2.rst

@@ -466,6 +466,43 @@ See :doc:`/build-customization` for more details.
 :Default: ``{}``
 
 
+build.commands
+``````````````
+
+Specify a list of commands that Read the Docs will run on the build process.
+When ``build.commands`` is used, none of the :term:`pre-defined build jobs` will be executed.
+(see :doc:`/build-customization` for more details).
+This allows you to run custom commands and control the build process completely.
+The ``_readthedocs/html`` directory (relative to the checkout's path) will be uploaded and hosted by Read the Docs.
+
+.. warning::
+
+   This feature is in a *beta phase* and could suffer incompatible changes or even removed completely in the near feature.
+   It does not yet support some of the Read the Docs' integrations like the :term:`flyout menu`, search and ads.
+   However, integrating all of them is part of the plan.
+   Use it under your own responsibility.
+
+.. code-block:: yaml
+
+   version: 2
+
+   build:
+     os: ubuntu-22.04
+     tools:
+       python: "3.10"
+     commands:
+       - pip install pelican
+       - pelican --settings docs/pelicanconf.py --output _readthedocs/html/ docs/
+
+.. note::
+
+   ``build.os`` and ``build.tools`` are also required when using ``build.commands``.
+
+:Type: ``list``
+:Required: ``false``
+:Default: ``[]``
+
+
 sphinx
 ~~~~~~
 

readthedocs/builds/migrations/0044_alter_version_documentation_type.py

@@ -0,0 +1,31 @@
+# Generated by Django 3.2.13 on 2022-05-30 10:24
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("builds", "0043_add_cancelled_state"),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="version",
+            name="documentation_type",
+            field=models.CharField(
+                choices=[
+                    ("sphinx", "Sphinx Html"),
+                    ("mkdocs", "Mkdocs"),
+                    ("sphinx_htmldir", "Sphinx HtmlDir"),
+                    ("sphinx_singlehtml", "Sphinx Single Page HTML"),
+                    ("mkdocs_html", "Mkdocs Html Pages"),
+                    ("generic", "Generic"),
+                ],
+                default="sphinx",
+                help_text="Type of documentation the version was built with.",
+                max_length=20,
+                verbose_name="Documentation type",
+            ),
+        ),
+    ]

readthedocs/config/config.py

@@ -11,6 +11,7 @@
 from django.conf import settings
 
 from readthedocs.config.utils import list_to_dict, to_dict
+from readthedocs.projects.constants import GENERIC
 
 from .find import find_one
 from .models import (
@@ -791,6 +792,11 @@ def validate_build_config_with_tools(self):
                     BuildJobs.__slots__,
                 )
 
+        commands = []
+        with self.catch_validation_error("build.commands"):
+            commands = self.pop_config("build.commands", default=[])
+            validate_list(commands)
+
         if not tools:
             self.error(
                 key='build.tools',
@@ -802,13 +808,25 @@ def validate_build_config_with_tools(self):
                 code=CONFIG_REQUIRED,
             )
 
+        if commands and jobs:
+            self.error(
+                key="build.commands",
+                message="The keys build.jobs and build.commands can't be used together.",
+                code=INVALID_KEYS_COMBINATION,
+            )
+
         build["jobs"] = {}
         for job, commands in jobs.items():
             with self.catch_validation_error(f"build.jobs.{job}"):
                 build["jobs"][job] = [
                     validate_string(command) for command in validate_list(commands)
                 ]
 
+        build["commands"] = []
+        for command in commands:
+            with self.catch_validation_error("build.commands"):
+                build["commands"].append(validate_string(command))
+
         build['tools'] = {}
         for tool, version in tools.items():
             with self.catch_validation_error(f'build.tools.{tool}'):
@@ -1293,6 +1311,7 @@ def build(self):
                 os=build['os'],
                 tools=tools,
                 jobs=BuildJobs(**build["jobs"]),
+                commands=build["commands"],
                 apt_packages=build["apt_packages"],
             )
         return Build(**build)
@@ -1326,6 +1345,9 @@ def mkdocs(self):
 
     @property
     def doctype(self):
+        if "commands" in self._config["build"] and self._config["build"]["commands"]:
+            return GENERIC
+
         if self.mkdocs:
             return 'mkdocs'
         return self.sphinx.builder

readthedocs/config/models.py

@@ -37,10 +37,11 @@ def __init__(self, **kwargs):
 
 class BuildWithTools(Base):
 
-    __slots__ = ("os", "tools", "jobs", "apt_packages")
+    __slots__ = ("os", "tools", "jobs", "apt_packages", "commands")
 
     def __init__(self, **kwargs):
-        kwargs.setdefault('apt_packages', [])
+        kwargs.setdefault("apt_packages", [])
+        kwargs.setdefault("commands", [])
         super().__init__(**kwargs)
 
 

readthedocs/config/tests/test_config.py

@@ -1013,6 +1013,59 @@ def test_new_build_config_conflict_with_build_python_version(self):
             build.validate()
         assert excinfo.value.key == 'python.version'
 
+    def test_commands_build_config(self):
+        build = self.get_build_config(
+            {
+                "build": {
+                    "os": "ubuntu-20.04",
+                    "tools": {"python": "3.8"},
+                    "commands": ["pip install pelican", "pelican content"],
+                },
+            },
+        )
+        build.validate()
+        assert isinstance(build.build, BuildWithTools)
+        assert build.build.commands == ["pip install pelican", "pelican content"]
+
+    def test_commands_build_config_invalid_command(self):
+        build = self.get_build_config(
+            {
+                "build": {
+                    "os": "ubuntu-20.04",
+                    "tools": {"python": "3.8"},
+                    "commands": "command as string",
+                },
+            },
+        )
+        with raises(InvalidConfig) as excinfo:
+            build.validate()
+        assert excinfo.value.key == "build.commands"
+
+    def test_commands_build_config_invalid_no_os(self):
+        build = self.get_build_config(
+            {
+                "build": {
+                    "commands": ["pip install pelican", "pelican content"],
+                },
+            },
+        )
+        with raises(InvalidConfig) as excinfo:
+            build.validate()
+        assert excinfo.value.key == "build.commands"
+
+    def test_commands_build_config_invalid_no_tools(self):
+        build = self.get_build_config(
+            {
+                "build": {
+                    "os": "ubuntu-22.04",
+                    "commands": ["pip install pelican", "pelican content"],
+                },
+            },
+        )
+        with raises(InvalidConfig) as excinfo:
+            build.validate()
+        assert excinfo.value.key == "build.tools"
+
     @pytest.mark.parametrize("value", ["", None, "pre_invalid"])
     def test_jobs_build_config_invalid_jobs(self, value):
         build = self.get_build_config(
@@ -2379,6 +2432,7 @@ def test_as_dict_new_build_config(self, tmpdir):
                         'full_version': settings.RTD_DOCKER_BUILD_SETTINGS['tools']['nodejs']['16'],
                     },
                 },
+                "commands": [],
                 "jobs": {
                     "pre_checkout": [],
                     "post_checkout": [],