Docs & FAQ (#318)

Docs!

* Add rendered docs for gh-pages delivery

Scheme based on
  sphinx-doc/sphinx#3382 (comment)

* Makefile support for gh-pages hosted docs

N.B. we require user to have sphinx installed globally at the moment.
That is somewhat reasonable, as Rust language uses sphinx for
documentation.

If we end up wanting specific sphinx extensions, we'll have to add
those requirements to a (not yet in existence) requirements-dev.txt.

Makefile

@@ -46,6 +46,10 @@ clean-cache: check_venv
 clean-python:
 	find . -type d -name venv -prune -o -type d -name __pycache__ -print0 | xargs -0 rm -rf
 
+doc-build: 
+	type sphinx-build || { echo "please install sphinx to build docs"; false; }
+	make -C docs html
+
 doctest: check_venv
 	pytest --doctest-modules -s --offline --debug-calls
 
@@ -69,6 +73,9 @@ install: venv
 setup_gsuite: check_venv
 	python -m bin.auth.setup_gsuite
 
+stage-docs: docs
+	git add --all --force docs/_build/html
+
 metatest:
 	pytest --aws-profiles example-account \
 		-o python_files=meta_test*.py \
@@ -89,6 +96,8 @@ build-image:
 	clean \
 	clean-cache \
 	clean-python \
+	doc-build \
 	flake8 \
 	install \
+	stage-docs \
 	venv

docs/Architecture.rst

@@ -0,0 +1,69 @@
+.. raw:: html
+
+   <!-- This Source Code Form is subject to the terms of the Mozilla Public
+      - License, v. 2.0. If a copy of the MPL was not distributed with this
+      - file, You can obtain one at https://mozilla.org/MPL/2.0/. -->
+
+.. _architecture:
+
+============
+Architecture
+============
+
+PyTest supports several different ways of organizing tests. For frost, we use a
+mixture of class based and declarative tests.
+
+In general the class holds session information, as PyTest treats class
+``__init__`` functions as session scoped fixtures. The class methods provide raw
+access to the service, and cache the result.
+
+Traditional PyTest fixtures (in ``conftest.py``) or "cache access functions" (in
+``resources.py``) are used to supply the data to tests. The tests are
+conventionally written in ``test_<foo>.py`` files, with a single function of the
+same name as the file. *(With "Black_", we stopped the tabs-vs-spaces debate,
+so redirected that energy to one-or-many-tests-per-file debate.)*
+
+A recommended way to organize your code is to create a directory per type of
+resource you test. E.g. ``aws/{elb,ec2,iam}/`` or
+``github/{orgs,branches,users}``. Whether it makes sense to have ``conftest.py``
+files at each level is up to the developer. There should only be one session
+client per service though.
+
+Caching
+=======
+
+.. note::
+   The caching operations is under consideration for deprecation. If you
+   intend to rely on caching, you should check with the active developers first.
+
+To implement caching:
+
+   #. Your class ``__init__`` method must accept and store a cache object.
+
+   #. Your data retrieval functions should be written to try the cache first
+         before fetching data from the service under test.
+
+   #. A cache_key global function is recommended as a means to ensure consistent
+         and non conflicting keys to store data in the cache. (The existing
+         functions tend to marshal the full data location path and arguments
+         into a string.)
+
+Expected Output flow
+====================
+
+Every test that fails needs to output sufficient information to allow downstream
+processes to take action on the failure (open an issue, or bug, or email the
+team or ...). All that information must be contained in the test id. Use the
+``ids`` argument to the ``pytest.mark.parametrize`` decorator to generate rich
+ids as needed. (See `PyTest docs`_.)
+
+A PyTest plugin in frost adds the option ``--json`` which outputs test failures
+as JSON objects which include the test's id, in addition to other context about
+the failed test. Using the ``--json`` option is the recommended way to provide
+actionalble data to processes further down the pipeline.
+
+The output flow will be installation specific.
+
+.. _PyTest docs: https://docs.pytest.org/en/stable/example/parametrize.html#paramexamples>`)
+
+.. _Black: https://black.readthedocs.io/

docs/CodingConventions.rst

@@ -0,0 +1,19 @@
+.. raw:: html
+
+   <!-- This Source Code Form is subject to the terms of the Mozilla Public
+      - License, v. 2.0. If a copy of the MPL was not distributed with this
+      - file, You can obtain one at https://mozilla.org/MPL/2.0/. -->
+
+===========
+Conventions
+===========
+
+- As mentioned elsewhere, all function *not* within a ``test_*.py`` file should
+  have doctest_ tests.
+
+- Frost tests are expected to support the ``--json`` option by ensuring ids used
+  in ``pytest.mark.parametrize`` contain sufficient information for downstream
+  processing.
+
+.. _doctest: https://docs.python.org/3.6/library/doctest.html
+

docs/FAQ.rst

@@ -0,0 +1,30 @@
+.. raw:: html
+
+   <!-- This Source Code Form is subject to the terms of the Mozilla Public
+      - License, v. 2.0. If a copy of the MPL was not distributed with this
+      - file, You can obtain one at https://mozilla.org/MPL/2.0/. -->
+
+==========================
+Frequently Asked Questions
+==========================
+
+Overview
+--------
+
+**What's the general flow of a "test"?**
+
+When you invoke a test, frost_ uses features of
+pytest_ to execute the test. Most commonly,
+the test will validate certain relationships about data files
+representing configuration data of some external service.
+
+If the data-under-test is already cached (and fresh enough), the cached
+data will be used. If the data is not available locally, pytest_
+fixtures are used to obtain or refresh the data required by that test.
+Any freshly retrieved data is cached for use by subsequent tests.
+
+This "lazy evaluation" of supplying data ensures the quickest possible
+turnaround time for ad-hoc queries.
+
+.. _pytest:  https://pytest.org/
+.. _frost: https://github.com/mozilla/frost

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/MozillaDeployment.rst

@@ -0,0 +1,29 @@
+.. raw:: html
+
+   <!-- This Source Code Form is subject to the terms of the Mozilla Public
+      - License, v. 2.0. If a copy of the MPL was not distributed with this
+      - file, You can obtain one at https://mozilla.org/MPL/2.0/. -->
+
+==================
+Mozilla Deployment
+==================
+
+Some details of the Mozilla deployment of Frost are listed here as an example of
+how it can be done.
+
+Frost jobs are run via Jenkins. Jobs are organized for both convenience and to
+accommodate different reporting intervals. Usually only a single service is
+queried in any particular job.
+
+The actual job runs in a docker container, which has the frost repository
+already checked out. Separate configuration repositories are also checked out at
+runtime, based on job parameters.
+
+Jobs have a common entry script, which performs any job-specific tasks before
+and after the main frost run. PyTest is always invoked with the ``--json``
+options supported by the frost extensions, and post processing steps are
+expected to use the JSON as input.
+
+_[The deployment is under revision. A rough "as is" doc may be found here__.]_
+
+__ https://docs.google.com/document/d/1ePUkJPcHEj9XxaVYr2TSABOxRjhDBKr2KSQ2EzgHJm4

docs/NewServices.rst

@@ -0,0 +1,71 @@
+.. raw:: html
+
+   <!-- This Source Code Form is subject to the terms of the Mozilla Public
+      - License, v. 2.0. If a copy of the MPL was not distributed with this
+      - file, You can obtain one at https://mozilla.org/MPL/2.0/. -->
+
+============================
+Adding a New Service
+============================
+Here are the steps to add a new service to the Frost framework.
+
+Claim a name
+============
+
+Like 'heroku' ;)
+
+Create a new directory by that name
+-----------------------------------
+
+Clone the repo::
+
+   git clone [email protected]:mozilla/frost.git
+   git checkout -b new_service
+
+Setup for new service::
+
+   mkdir heroku
+   cd heroku
+
+Create Default Files
+--------------------
+
+The new service should be a Python Package::
+
+   touch __init__.py client.py resources.py conftest.py
+
+Commit shell::
+
+   git add .
+   git push -m 'Adding new service Heroku'
+
+Add Service Specific Content
+============================
+
+``client.py``: responsible for obtaining data from the service, and
+placing it into the PyTest cache. The client module typically exposes the data via a
+"{service}_client" object. The PyTest framework will instantiate the client
+before any tests are run with all the configuration & credential
+information provided by the configuration files and command line
+options. (See :ref:`architecture` for status of cache functionality.)
+
+``resources.py``: holds mapping functions which convert the data from
+the cache into the format expected by the tests. This should be the only
+file which imports the instantiation of the client. (Future best
+practices may pre-populate the cache outside of the PyTest execution.)
+
+``conftest.py`` (optional) As much as possible, put service specific
+options, etc. in this local file. (Some things may not work c.f. BUG.)
+In conventional ``PyTest`` usage, ``conftest.py`` would contain fixture
+routines which did the combined steps of fetching the data and providing to the
+tests.  If caching is not important, the traditional approach may be used.
+
+Tests for these support files should be included as doc tests whenever
+practical. If possible, the default of executing the module should be to run
+the doc tests.
+
+Add Service Specific Tests
+--------------------------
+
+``test_*.py``: normal PyTest tests, which each import the resources they
+need to test.

docs/_build/html/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 07c966ec7863c8a9e214c5ee2bd399ef
+tags: 645f666f9bcd5a90fca523b33c5a78b7