Docs: Split Custom Domains as Explanation and How-to Guide (Diátaxis)

docs/conf.py

@@ -32,6 +32,7 @@
     "multiproject",
     "sphinx.ext.autosectionlabel",
     "sphinx.ext.autodoc",
+    "sphinx.ext.extlinks",
     "sphinx.ext.intersphinx",
     "sphinxcontrib.httpdomain",
     "sphinxcontrib.video",
@@ -159,8 +160,11 @@
 hoverxref_domains = ["py"]
 hoverxref_roles = [
     "option",
-    "doc",  # Documentation pages
-    "term",  # Glossary terms
+    # Documentation pages
+    # Not supported yet: https://github.com/readthedocs/sphinx-hoverxref/issues/18
+    "doc",
+    # Glossary terms
+    "term",
 ]
 hoverxref_role_types = {
     "mod": "modal",  # for Python Sphinx Domain
@@ -206,3 +210,7 @@
     # This page is under login
     r"https://readthedocs\.org/accounts/gold",
 ]
+
+extlinks = {
+    "rtd-issue": ("https://github.com/readthedocs/readthedocs.org/issues/%s", "#"),
+}

docs/user/canonical-urls.rst

@@ -3,21 +3,34 @@ Canonical URLs
 
 A `canonical URL`_
 allows you to specify the preferred version of a web page to prevent duplicated content.
-They are mainly used by search engines to link users to the correct
-version and domain of your documentation.
+Here are some examples of when a canonical URL is used:
 
-If canonical URL's aren't used,
+- Search engines use your canonical URL to link users to the correct version and domain of your documentation.
+- Many popular chat clients and social media networks generate link previews,
+  using your canonical URL as the final destination.
+
+If canonical URLs aren't used,
 it's easy for outdated documentation to be the top search result for various pages in your documentation.
 This is not a perfect solution for this problem,
 but generally people finding outdated documentation is a big problem,
 and this is one of the suggested ways to solve it from search engines.
 
 .. _canonical URL: https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls
 
+.. tip::
+
+   In most cases, Read the Docs will automatically generate a canonical URL for Sphinx projects.
+   Most Sphinx users do not need to take further action.
+
+.. seealso::
+
+   :doc:`/guides/canonical-urls`
+      More information on how to enable canonical URLs in your project.
+
 How Read the Docs generates canonical URLs
 ------------------------------------------
 
-The canonical URL takes into account:
+The canonical URL takes the following into account:
 
 * The default version of your project (usually "latest" or "stable").
 * The canonical :doc:`custom domain </custom-domains>` if you have one,
@@ -40,32 +53,10 @@ thus avoiding duplicating the content.
 Implementation
 --------------
 
-The canonical URL is set in HTML with a ``link`` element.
-For example, this page has a canonical URL of:
+A canonical URL is automatically specified in the HTML output with a ``<link>`` element.
+For instance, regardless of whether you are viewing this page on ``/en/latest`` or ``/en/stable``,
+the following HTML header data will be present:
 
 .. code-block:: html
 
    <link rel="canonical" href="https://docs.readthedocs.io/en/stable/canonical-urls.html" />
-
-Sphinx
-~~~~~~
-
-If you are using :doc:`Sphinx </intro/getting-started-with-sphinx>`,
-Read the Docs will set the value of the html_baseurl_ setting (if isn't already set) to your canonical domain.
-If you already have ``html_baseurl`` set, you need to ensure that the value is correct.
-
-.. _html_baseurl: https://www.sphinx-doc.org/page/usage/configuration.html#confval-html_baseurl
-
-Mkdocs
-~~~~~~
-
-For :doc:`MkDocs </intro/getting-started-with-mkdocs>` this isn't done automatically,
-but you can use the site_url_ setting to set a similar value.
-
-.. _site_url: https://www.mkdocs.org/user-guide/configuration/#site_url
-
-.. warning::
-
-   If you change your default version or canonical domain,
-   you'll need to re-build all your versions in order to update their
-   canonical URL to the new one.

docs/user/custom-domains.rst

@@ -1,126 +1,101 @@
 Custom Domains
 ==============
 
-Custom domains allow you to serve your documentation from your own domain.
-This is great for maintaining a consistent brand for your documentation and application.
+You can serve your documentation project from your own domain,
+for instance ``docs.example.com``.
+This is great for maintaining a consistent brand for your product and its documentation.
 
-By default, your documentation is served from a Read the Docs :ref:`subdomain <hosting:subdomain support>` using the project's :term:`slug`:
+*By default*, your documentation is served from a Read the Docs :ref:`subdomain <hosting:subdomain support>` using the project's :term:`slug`:
 
 * ``<slug>.readthedocs.io`` for |org_brand|
 * ``<slug>.readthedocs-hosted.com`` for |com_brand|.
 
-For example if you import your project and it gets the :term:`slug` ``example-docs``, it will be served from ``https://example-docs.readthedocs.io``.
+.. seealso::
 
-.. contents:: Contents
-    :local:
+    :doc:`/guides/custom-domains`
+        Information on creating and managing custom domains,
+        and common configurations you might use to set up your domain
 
-Adding a custom domain
-----------------------
+How custom domains work
+-----------------------
 
-To setup your custom domain, follow these steps:
+To use a custom domain, two actions are needed from you:
 
-#. Go the :guilabel:`Admin` tab of your project.
-#. Click on :guilabel:`Domains`.
-#. Enter the domain where you want to serve the documentation from (e.g. ``docs.example.com``).
-#. Mark the :guilabel:`Canonical` option if you want use this domain
-   as your :doc:`canonical domain </canonical-urls>`.
-#. Click on :guilabel:`Add`.
-#. At the top of the next page you'll find the value of the DNS record that you need to point your domain to.
-   For |org_brand| this is ``readthedocs.io``, and for :doc:`/commercial/index`
-   the record is in the form of ``<hash>.domains.readthedocs.com``.
+#.  Enter the domain in your Read the Docs project's :guilabel:`Admin`
+#.  Update your DNS provider with a new DNS entry. The name and value of the DNS entry is found in Read the Docs' :guilabel:`Admin`.
 
-   .. note::
+Once the new DNS record has propagated,
+Read the Docs automatically issues an SSL certificate through Cloudflare and starts serving your documentation.
 
-      For a subdomain like ``docs.example.com`` add a CNAME record,
-      and for a root domain like ``example.com`` use an ANAME or ALIAS record.
+.. image:: img/mermaid-custom-domains.png
+   :alt: Diagram of the process of adding a custom domain on Read the Docs
 
-By default, we provide a validated SSL certificate for the domain,
-managed by `Cloudflare <https://www.cloudflare.com/>`_.
-The SSL certificate issuance should happen within a few minutes,
-but might take up to one hour.
-See `SSL certificate issue delays`_ for more troubleshooting options.
+..
+   We have generated an PNG version of the following diagram using mermaid.live
+   Firstly, we generate an SVG, then we render it in a browser, then we take a screenshot,
+   then we paste it into GIMP or similar and make the background transparent.
 
-As an example, our blog's DNS record looks like this:
+   If you wish to sketch diagrams locally, you can add sphinxcontrib-mermaid to
+   this project's extensions and keep using the below code.
 
-.. prompt:: bash $, auto
+   SVG does not work because it embeds fontawesome from CDN (which is blocked by CSP)
 
-   $ dig +short CNAME blog.readthedocs.com
-     readthedocs.io.
+   PLEASE KEEP THIS SOURCE CODE UPDATED
+   .. mermaid::
 
-.. warning::
+       graph TD
+           subgraph rtd [On Read the Docs]
+             A(fa:fa-pencil Add docs.example.com as Custom Domain)
+           end
+           subgraph dns [On your domain's DNS administration]
+             B(fa:fa-pencil Edit/add a DNS entry for docs.example.com<br>making it point to Read the Docs)
+           end
 
-   We don't support pointing subdomains or root domains to a project using A records.
-   DNS A records require a static IP address and our IPs may change without notice.
+           rtd & dns-->C(fa:fa-spinner Wait for DNS propagation.<br>Usually just a few minutes)
 
+           direction LR
+           subgraph automatic [fa:fa-paper-plane The rest is handled automatically]
+             direction TB
+             D(fa:fa-spinner The next time your project is built,<br>its Canonical URLs use docs.example.com)
+             D-->E(Visit https://docs.example.com)
+             E-->F(fa:fa-lock Correct SSL Certificate <br>automatically used)
+             F-->G(fa:fa-check Read the Docs knows<br> to serve your project <br>at docs.example.com)
+           end
 
-Removing a custom domain
-------------------------
+           C-->automatic
 
-To remove a custom domain:
 
-#. Go the :guilabel:`Admin` tab of your project.
-#. Click on :guilabel:`Domains`.
-#. Click the :guilabel:`Remove` button next to the domain.
-#. Click :guilabel:`Confirm` on the confirmation page.
+Your documentation can have multiple secondary domains but only one **canonical** domain name.
+Additional domains or subdomains will redirect to the canonical domain.
 
-.. warning::
+To make this work, Read the Docs generates a special text that you are responsible for copy-pasting to your domain's DNS.
+In most cases, the ``CNAME`` record is used.
+This is all that's needed for a web browser to resolve your domain name to Read the Docs' servers and for our servers to match the right documentation project.
+You can find step-by-step instructions for this in :doc:`/guides/custom-domains`.
 
-    Once a domain is removed,
-    your previous documentation domain is no longer served by Read the Docs,
-    and any request for it will return a 404 Not Found!
+Read the Docs uses a :ref:`hosting:Content Delivery Network (CDN)` to host and serve your documentation pages.
+This final step isn't changed by a custom domain
+and therefore the response times are unaffected as the delivery of resources happens through the same CDN setup.
 
-Strict Transport Security (HSTS) and other custom headers
----------------------------------------------------------
+Considerations for custom domain usage
+--------------------------------------
 
-By default, we do not return a `Strict Transport Security header`_ (HSTS) for user custom domains.
-This is a conscious decision as it can be misconfigured in a not easily reversible way.
-For both |org_brand| and |com_brand|, HSTS and other custom headers can be set upon request.
+Some open source projects have seen their domains expire.
+Even prominent ones.
+**It's important that you give the responsibility for managing your domain to someone reliable in your organization.**
 
-We always return the HSTS header with a max-age of at least one year
-for our own domains including ``*.readthedocs.io``, ``*.readthedocs-hosted.com``, ``readthedocs.org`` and ``readthedocs.com``.
+The **canonical domain** feature allows you to have several domains and the canonical domain will be indexed by search engines.
+The domain that you choose as your canonical domain is by far the most important one.
+If you lose the canonical domain,
+someone else can set up a website that search results will end up referring to.
 
-Please contact :doc:`support` if you want to add a custom header to your domain.
+.. seealso::
 
-.. _Strict Transport Security header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security
+   In a URL, both the domain and the path (``https://<domain>/<path>``) are important.
+   In combination, they are referred to as the *canonical URL* of a resource.
 
-Multiple documentation sites as sub-folders of a domain
--------------------------------------------------------
+   Most documentation projects are versioned.
+   Therefore, it's important to ensure that incoming links and search engine results point to the canonical URL of the resource
+   and not a specific version that becomes outdated.
 
-You may host multiple documentation repositories as **sub-folders of a single domain**.
-For example, ``docs.example.org/projects/repo1`` and ``docs.example.org/projects/repo2``.
-This is `a way to boost the SEO of your website <https://moz.com/blog/subdomains-vs-subfolders-rel-canonical-vs-301-how-to-structure-links-optimally-for-seo-whiteboard-friday>`_.
-
-See :doc:`subprojects` for more information.
-
-Troubleshooting
----------------
-
-SSL certificate issue delays
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The status of your domain validation and certificate can always be seen on the details page for your domain
-under :guilabel:`Admin` > :guilabel:`Domains` > :guilabel:`YOURDOMAIN.TLD (details)`.
-
-Domains are usually validated and a certificate issued within minutes.
-However, if you setup the domain in Read the Docs without provisioning the necessary DNS changes
-and then update DNS hours or days later,
-this can cause a delay in validating because there is an exponential back-off in validation.
-
-.. tip::
-
-    Loading the domain details in the Read the Docs dashboard and saving the domain again will force a revalidation.
-
-The validation process period has ended
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-After you add a new custom domain, you have 30 days to complete the configuration.
-Once that period has ended, we will stop trying to validate your domain.
-If you still want to complete the configuration,
-go to your domain and click on :guilabel:`Save` to restart the process.
-
-Migrating from GitBook
-~~~~~~~~~~~~~~~~~~~~~~
-
-If your custom domain was previously used in GitBook, contact GitBook support (via live chat in their website)
-to remove the domain name from their DNS Zone in order for your domain name to work with Read the Docs,
-else it will always redirect to GitBook.
+   To learn more about canonical URLs, see: :doc:`/canonical-urls`

docs/user/glossary.rst

@@ -31,6 +31,12 @@ Glossary
       and hypens. You can retreive your project or version slugs from
       :doc:`our API <api/v3>`.
 
+   subproject
+      Project A can be configured such that when requesting a URL ``/projects/<subproject-slug>``,
+      the root of project B is returned.
+      In this case, *project B* is the subproject.
+      Read more in :doc:`/subprojects`.
+
    root URL
       Home URL of your documentation without the ``/<lang>`` and ``/<version>`` segments.
       For projects without custom domains, the one ending in ``.readthedocs.io/``

docs/user/guides/administrators.rst

@@ -12,6 +12,8 @@ have a look at our :doc:`/tutorial/index`.
 .. toctree::
    :maxdepth: 1
 
+   Manage Custom Domains <custom-domains>
+   Enable Canonical URLs <canonical-urls>
    technical-docs-seo-guide
    manage-translations-sphinx
    hiding-a-version

docs/user/guides/canonical-urls.rst

@@ -0,0 +1,58 @@
+How to Enable Canonical URLs
+============================
+
+In this guide, we introduce relevant settings for enabling canonical URLs in popular documentation frameworks.
+
+If you need to customize the domain from which your documentation project is served,
+please refer to :doc:`/guides/custom-domains`.
+
+Sphinx
+~~~~~~
+
+If you are using :doc:`Sphinx </intro/getting-started-with-sphinx>`,
+Read the Docs will automatically add a default value of the html_baseurl_ setting matching your canonical domain.
+
+If you are using a custom ``html_baseurl`` in your ``conf.py``,
+you have to ensure that the value is correct.
+This can be complex,
+supporting pull request builds (which are published on a separate domain),
+special branches
+or if you are using :term:`subproject` s or :ref:`translations <localization:Localization of Documentation>`.
+We recommend not including a ``html_baseurl`` in your ``conf.py``,
+and letting Read the Docs define it.
+
+.. _html_baseurl: https://www.sphinx-doc.org/page/usage/configuration.html#confval-html_baseurl
+
+MkDocs
+~~~~~~
+
+For :doc:`MkDocs </intro/getting-started-with-mkdocs>` we do not define your canonical domain automatically,
+but you can use the site_url_ setting to set a similar value.
+
+In your ``mkdocs.yml``, define the following:
+
+.. code-block:: yaml
+
+   # Canonical URL, adjust as need with respect to your slug, language,
+   # default branch and if you use a custom domain.
+   site_url: https://<slug>.readthedocs.io/en/stable/
+
+Note that this will define the same canonical URL for all your branches and versions.
+According to MkDocs, defining site_url_ will only define the canonical URL of a website and does not affect the base URL of generated links, CSS, or Javascript files.
+
+.. note::
+
+   2 known issues are currently making it impossible to use `environment variables in MkDocs configuration`_.
+   Once these issues are solved, it will be easier.
+
+   - Support for ``!ENV``: :rtd-issue:`8529`
+   - Add environment variable for canonical URL: :rtd-issue:`9781`
+
+.. _site_url: https://www.mkdocs.org/user-guide/configuration/#site_url
+.. _environment variables in MkDocs configuration: https://www.mkdocs.org/user-guide/configuration/#environment-variables
+
+.. warning::
+
+   If you change your default version or canonical domain,
+   you'll need to re-build all your versions in order to update their
+   canonical URL to the new one.