Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it easier to configure docsite builds, allow to override collection URLs and installation instructions #26

Merged
merged 4 commits into from
Aug 20, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions changelogs/fragments/26-sphinx-antsibull-cfg.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
minor_changes:
- "The ``sphinx-init`` subcommand now also creates an ``antsibull-docs.cfg`` file and moves configuration settings from CLI flags in ``build.sh`` to this configuration file (https://github.com/ansible-community/antsibull-docs/pull/26)."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

excellent, this should make for some interesting possibilities in https://github.com/ansible-community/github-docs-build/ maybe 🤔

- "There are two new options for explicitly specified configuration files named ``collection_url`` and ``collection_install``.
These allow to override the URLs pointing to collections (default link to galaxy.ansible.com),
and the commands to install collections (use ``ansible-galaxy collection install`` by default).
This can be useful when documenting (internal) collections that are not available on Ansible Galaxy.
The default ``antsibull-docs.cfg`` generated by the ``sphinx-init`` subcommand shows how this can be configured
(https://github.com/ansible-community/antsibull-docs/issues/15, https://github.com/ansible-community/antsibull-docs/pull/26)."
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ antsibull-docs = "antsibull_docs.cli.antsibull_docs:main"
[tool.poetry.dependencies]
python = "^3.6.1"
ansible-pygments = "*"
antsibull-core = ">= 1.0.0, < 2.0.0"
antsibull-core = ">= 1.2.0, < 2.0.0"
asyncio-pool = "*"
docutils = "*"
jinja2 = "*"
Expand Down
14 changes: 14 additions & 0 deletions src/antsibull_docs/app_context.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Author: Felix Fontein <[email protected]>
# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or
# https://www.gnu.org/licenses/gpl-3.0.txt)
# SPDX-License-Identifier: GPL-3.0-or-later
# SPDX-FileCopyrightText: 2022, Ansible Project
"""Local app and lib context provider"""

# pylint: disable-next=unused-import
from antsibull_core.app_context import lib_ctx # noqa
from antsibull_core.app_context import AppContextWrapper
from antsibull_docs.schemas.app_context import DocsAppContext


app_ctx: AppContextWrapper[DocsAppContext] = AppContextWrapper()
15 changes: 10 additions & 5 deletions src/antsibull_docs/cli/antsibull_docs.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,12 @@
InvalidArgumentError, get_toplevel_parser, normalize_toplevel_options
)
from antsibull_core.compat import BooleanOptionalAction # noqa: E402
from antsibull_core.config import load_config # noqa: E402
from antsibull_core.config import ConfigError, load_config # noqa: E402
from antsibull_core.filesystem import UnableToCheck, writable_via_acls # noqa: E402

from ..constants import DOCUMENTABLE_PLUGINS # noqa: E402
from ..docs_parsing.fqcn import is_fqcn # noqa: E402
from ..schemas.app_context import DocsAppContext # noqa: E402
from .doc_commands import ( # noqa: E402
collection, current, devel, plugin, stable, sphinx_init, lint_collection_docs
)
Expand Down Expand Up @@ -421,15 +422,19 @@ def run(args: List[str]) -> int:
try:
parsed_args: argparse.Namespace = parse_args(program_name, args[1:])
except InvalidArgumentError as e:
flog.error(e)
print(e)
return 2
flog.fields(args=parsed_args).info('Arguments parsed')

cfg = load_config(parsed_args.config_file)
flog.fields(config=cfg).info('Config loaded')
try:
cfg = load_config(parsed_args.config_file, app_context_model=DocsAppContext)
flog.fields(config=cfg).info('Config loaded')
except ConfigError as e:
print(e)
return 2

context_data = app_context.create_contexts(args=parsed_args, cfg=cfg)
context_data = app_context.create_contexts(
args=parsed_args, cfg=cfg, app_context_model=DocsAppContext)
with app_context.app_and_lib_context(context_data) as (app_ctx, dummy_):
twiggy.dict_config(app_ctx.logging_cfg.dict())
flog.debug('Set logging config')
Expand Down
2 changes: 1 addition & 1 deletion src/antsibull_docs/cli/doc_commands/collection.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,14 @@
import aiohttp
import asyncio_pool # type: ignore[import]

from antsibull_core import app_context
from antsibull_core.collections import install_together
from antsibull_core.compat import asyncio_run
from antsibull_core.galaxy import CollectionDownloader
from antsibull_core.logging import log
from antsibull_core.venv import FakeVenvRunner

from .stable import generate_docs_for_all_collections
from ... import app_context

if t.TYPE_CHECKING:
import semantic_version as semver # pylint:disable=unused-import
Expand Down
2 changes: 1 addition & 1 deletion src/antsibull_docs/cli/doc_commands/current.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
# SPDX-FileCopyrightText: 2020, Ansible Project
"""Entrypoint to the antsibull-docs script."""

from antsibull_core import app_context
from antsibull_core.logging import log
from antsibull_core.venv import FakeVenvRunner

from .stable import generate_docs_for_all_collections
from ... import app_context


mlog = log.fields(mod=__name__)
Expand Down
2 changes: 1 addition & 1 deletion src/antsibull_docs/cli/doc_commands/devel.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
import aiohttp
import asyncio_pool # type: ignore[import]

from antsibull_core import app_context
from antsibull_core.ansible_core import get_ansible_core
from antsibull_core.collections import install_together
from antsibull_core.compat import asyncio_run
Expand All @@ -24,6 +23,7 @@
from antsibull_core.venv import VenvRunner

from .stable import generate_docs_for_all_collections
from ... import app_context

if t.TYPE_CHECKING:
import semantic_version as semver # pylint:disable=unused-import
Expand Down
10 changes: 8 additions & 2 deletions src/antsibull_docs/cli/doc_commands/lint_collection_docs.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,13 @@
# SPDX-FileCopyrightText: 2020, Ansible Project
"""Entrypoint to the antsibull-docs script."""

from antsibull_core import app_context
from antsibull_core.logging import log

from ... import app_context
from ...collection_links import lint_collection_links
from ...lint_extra_docs import lint_collection_extra_docs_files
from ...lint_plugin_docs import lint_collection_plugin_docs
from ...utils.collection_name_transformer import CollectionNameTransformer


mlog = log.fields(mod=__name__)
Expand Down Expand Up @@ -39,7 +40,12 @@ def lint_collection_docs() -> int:

if plugin_docs:
flog.notice('Linting plugin docs')
errors.extend(lint_collection_plugin_docs(collection_root))
collection_url = CollectionNameTransformer(
app_ctx.collection_url, 'https://galaxy.ansible.com/{namespace}/{name}')
collection_install = CollectionNameTransformer(
app_ctx.collection_install, 'ansible-galaxy collection install {namespace}.{name}')
errors.extend(lint_collection_plugin_docs(
collection_root, collection_url=collection_url, collection_install=collection_install))

messages = sorted(set(f'{error[0]}:{error[1]}:{error[2]}: {error[3]}' for error in errors))

Expand Down
12 changes: 10 additions & 2 deletions src/antsibull_docs/cli/doc_commands/plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,19 @@

import sh

from antsibull_core import app_context
from antsibull_core.compat import asyncio_run
from antsibull_core.logging import log
from antsibull_core.vendored.json_utils import _filter_non_json_lines
from antsibull_core.venv import FakeVenvRunner

from .stable import normalize_plugin_info
from ... import app_context
from ...augment_docs import augment_docs
from ...collection_links import CollectionLinks
from ...docs_parsing import AnsibleCollectionMetadata
from ...docs_parsing.fqcn import get_fqcn_parts, is_fqcn
from ...jinja2.environment import doc_environment
from ...utils.collection_name_transformer import CollectionNameTransformer
from ...write_docs import write_plugin_rst


Expand Down Expand Up @@ -100,7 +101,14 @@ def generate_plugin_docs(plugin_type: str, plugin_name: str,
}))

# Setup the jinja environment
env = doc_environment(('antsibull_docs.data', 'docsite'))
collection_url = CollectionNameTransformer(
app_ctx.collection_url, 'https://galaxy.ansible.com/{namespace}/{name}')
collection_install = CollectionNameTransformer(
app_ctx.collection_install, 'ansible-galaxy collection install {namespace}.{name}')
env = doc_environment(
('antsibull_docs.data', 'docsite'),
collection_url=collection_url,
collection_install=collection_install)
# Get the templates
plugin_tmpl = env.get_template('plugin.rst.j2')
error_tmpl = env.get_template('plugin-error.rst.j2')
Expand Down
25 changes: 23 additions & 2 deletions src/antsibull_docs/cli/doc_commands/sphinx_init.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,11 @@

import os
import os.path
import typing as t

from antsibull_core import app_context
from antsibull_core.logging import log

from ... import app_context
from ...jinja2.environment import doc_environment


Expand All @@ -19,6 +20,7 @@

TEMPLATES = [
'.gitignore',
'antsibull-docs.cfg',
'build.sh',
'conf.py',
'requirements.txt',
Expand All @@ -42,6 +44,18 @@ def write_file(filename: str, content: str) -> None:
f.write(content)


def toperky(value: t.Any) -> str:
if isinstance(value, str):
value = value.replace('\\', '\\\\')
value = value.replace('\n', '\\n')
value = value.replace('\r', '\\r')
value = value.replace('"', '\\"')
if all(ch not in value for ch in '\\={}[]') and value.strip() == value:
return value
return f'"{value}"'
raise Exception(f'toperky filter cannot handle type {type(value)}')


def site_init() -> int:
"""
Initialize a Sphinx site template for a collection docsite.
Expand All @@ -67,14 +81,19 @@ def site_init() -> int:
use_html_blobs = app_ctx.use_html_blobs
breadcrumbs = app_ctx.breadcrumbs
indexes = app_ctx.indexes
collection_url = app_ctx.collection_url
collection_install = app_ctx.collection_install

sphinx_theme = 'sphinx_ansible_theme'
sphinx_theme_package = 'sphinx-ansible-theme >= 0.9.0'
if app_ctx.extra['sphinx_theme'] != 'sphinx-ansible-theme':
sphinx_theme = app_ctx.extra['sphinx_theme']
sphinx_theme_package = app_ctx.extra['sphinx_theme']

env = doc_environment(('antsibull_docs.data', 'sphinx_init'))
env = doc_environment(
('antsibull_docs.data', 'sphinx_init'),
extra_filters={'toperky': toperky},
)

for filename in TEMPLATES:
source = filename.replace('.', '_').replace('/', '_') + '.j2'
Expand All @@ -93,6 +112,8 @@ def site_init() -> int:
indexes=indexes,
sphinx_theme=sphinx_theme,
sphinx_theme_package=sphinx_theme_package,
collection_url=collection_url,
collection_install=collection_install,
) + '\n'

destination = os.path.join(dest_dir, filename)
Expand Down
28 changes: 24 additions & 4 deletions src/antsibull_docs/cli/doc_commands/stable.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
import asyncio_pool # type: ignore[import]
from pydantic import ValidationError

from antsibull_core import app_context
from antsibull_core.ansible_core import get_ansible_core
from antsibull_core.collections import install_together
from antsibull_core.compat import asyncio_run, best_get_loop
Expand All @@ -27,6 +26,7 @@
from antsibull_core.logging import log
from antsibull_core.venv import VenvRunner, FakeVenvRunner

from ... import app_context
from ...augment_docs import augment_docs
from ...extra_docs import load_collections_extra_docs
from ...collection_links import load_collections_links
Expand All @@ -38,6 +38,7 @@
remove_redirect_duplicates,
)
from ...schemas.docs import DOCS_SCHEMAS
from ...utils.collection_name_transformer import CollectionNameTransformer
from ...write_docs import (
output_all_plugin_rst,
output_all_plugin_stub_rst,
Expand Down Expand Up @@ -333,6 +334,8 @@ def generate_docs_for_all_collections(venv: t.Union[VenvRunner, FakeVenvRunner],
flog = mlog.fields(func='generate_docs_for_all_collections')
flog.notice('Begin')

app_ctx = app_context.app_ctx.get()

# Get the info from the plugins
plugin_info, collection_metadata = asyncio_run(get_ansible_plugin_info(
venv, collection_dir, collection_names=collection_names))
Expand Down Expand Up @@ -383,21 +386,34 @@ def generate_docs_for_all_collections(venv: t.Union[VenvRunner, FakeVenvRunner],

collection_namespaces = get_collection_namespaces(collection_to_plugin_info.keys())

collection_url = CollectionNameTransformer(
app_ctx.collection_url, 'https://galaxy.ansible.com/{namespace}/{name}')
collection_install = CollectionNameTransformer(
app_ctx.collection_install, 'ansible-galaxy collection install {namespace}.{name}')

# Only build top-level index if requested
if create_indexes:
asyncio_run(output_collection_index(
collection_to_plugin_info, collection_namespaces, dest_dir, breadcrumbs=breadcrumbs,
for_official_docsite=for_official_docsite))
asyncio_run(output_collection_index(collection_to_plugin_info, collection_namespaces,
dest_dir, collection_url=collection_url,
collection_install=collection_install,
breadcrumbs=breadcrumbs,
for_official_docsite=for_official_docsite))
flog.notice('Finished writing collection index')
asyncio_run(output_collection_namespace_indexes(collection_namespaces, dest_dir,
collection_url=collection_url,
collection_install=collection_install,
breadcrumbs=breadcrumbs,
for_official_docsite=for_official_docsite))
flog.notice('Finished writing collection namespace index')
asyncio_run(output_plugin_indexes(plugin_contents, dest_dir,
collection_url=collection_url,
collection_install=collection_install,
for_official_docsite=for_official_docsite))
flog.notice('Finished writing plugin indexes')

asyncio_run(output_indexes(collection_to_plugin_info, dest_dir,
collection_url=collection_url,
collection_install=collection_install,
collection_metadata=collection_metadata,
squash_hierarchy=squash_hierarchy,
extra_docs_data=extra_docs_data,
Expand All @@ -407,6 +423,8 @@ def generate_docs_for_all_collections(venv: t.Union[VenvRunner, FakeVenvRunner],
flog.notice('Finished writing indexes')

asyncio_run(output_all_plugin_stub_rst(stubs_info, dest_dir,
collection_url=collection_url,
collection_install=collection_install,
collection_metadata=collection_metadata,
link_data=link_data,
squash_hierarchy=squash_hierarchy,
Expand All @@ -415,6 +433,8 @@ def generate_docs_for_all_collections(venv: t.Union[VenvRunner, FakeVenvRunner],

asyncio_run(output_all_plugin_rst(collection_to_plugin_info, new_plugin_info,
nonfatal_errors, dest_dir,
collection_url=collection_url,
collection_install=collection_install,
collection_metadata=collection_metadata,
link_data=link_data,
squash_hierarchy=squash_hierarchy,
Expand Down
2 changes: 1 addition & 1 deletion src/antsibull_docs/data/docsite/plugin-error.rst.j2
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ The errors were:

{% endfor %}

File a bug with the `@{ collection }@ collection <https://galaxy.ansible.com/@{collection | replace(".", "/", 1)}@>`_ in order to have it corrected.
File a bug with the `@{ collection }@ collection <@{ collection | collection_url }@>`_ in order to have it corrected.
2 changes: 1 addition & 1 deletion src/antsibull_docs/data/docsite/plugin-redirect.rst.j2
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
the same module name.
{% else %}
.. note::
This redirect is part of the `@{collection}@ collection <https://galaxy.ansible.com/@{collection | replace('.', '/', 1)}@>`_{% if collection_version %} (version @{ collection_version }@){% endif %}.
This redirect is part of the `@{collection}@ collection <@{ collection | collection_url }@>`_{% if collection_version %} (version @{ collection_version }@){% endif %}.

{% if not deprecation %}
To use it in a playbook, specify: :code:`@{plugin_name}@`.
Expand Down
2 changes: 1 addition & 1 deletion src/antsibull_docs/data/docsite/plugin-tombstone.rst.j2
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
This module was part of ``ansible-core`` and was included in all Ansible installations.
{% else %}
.. note::
This plugin was part of the `@{collection}@ collection <https://galaxy.ansible.com/@{collection | replace('.', '/', 1)}@>`_{% if collection_version %} (version @{ collection_version }@){% endif %}.
This plugin was part of the `@{collection}@ collection <@{ collection | collection_url }@>`_{% if collection_version %} (version @{ collection_version }@){% endif %}.
{% endif %}

This {% if plugin_type == 'module' %}module{% else %}@{ plugin_type }@ plugin{% endif %} has been removed
Expand Down
4 changes: 2 additions & 2 deletions src/antsibull_docs/data/docsite/plugin.rst.j2
Original file line number Diff line number Diff line change
Expand Up @@ -91,15 +91,15 @@
the same {% if plugin_type == 'module' %}module{% else %}@{ plugin_type }@ plugin{% endif %} name.
{% else %}
.. note::
This {% if plugin_type == 'module' %}module{% else %}@{ plugin_type }@ plugin{% endif %} is part of the `@{collection}@ collection <https://galaxy.ansible.com/@{collection | replace('.', '/', 1)}@>`_{% if collection_version %} (version @{ collection_version }@){% endif %}.
This {% if plugin_type == 'module' %}module{% else %}@{ plugin_type }@ plugin{% endif %} is part of the `@{collection}@ collection <@{ collection | collection_url }@>`_{% if collection_version %} (version @{ collection_version }@){% endif %}.
{% if for_official_docsite %}

You might already have this collection installed if you are using the ``ansible`` package.
It is not included in ``ansible-core``.
To check whether it is installed, run :code:`ansible-galaxy collection list`.
{% endif %}

To install it, use: :code:`ansible-galaxy collection install @{collection}@`.
To install it, use: @{ collection | collection_install | rst_code }@.
{% if doc['requirements'] %}
You need further requirements to be able to use this {% if plugin_type == 'module' %}module{% else %}@{ plugin_type }@ plugin{% endif %},
see :ref:`Requirements <ansible_collections.@{plugin_name}@_@{plugin_type}@_requirements>` for details.
Expand Down
Loading