feat: add docker-compose and revamp docs

Add's docker-compose YAML config file and boto config for use with
docker-compose.

Major refactor to documentation to break-down docs by user
audience.

Closes #559

autopush/logging.py

@@ -55,6 +55,12 @@
 
 @implementer(ILogObserver)
 class PushLogger(object):
+    """Twisted LogObserver implementation
+
+    Supports firehose delivery, Raven exception reporting, and json/test
+    console debugging output.
+
+    """
     def __init__(self, logger_name, log_level="debug", log_format="json",
                  log_output="stdout", sentry_dsn=None,
                  firehose_delivery_stream=None):
@@ -175,6 +181,7 @@ def setup_logging(cls, logger_name, log_level="info", log_format="json",
 
 
 class FirehoseProcessor(object):
+    """Batches log events for sending to AWS FireHose"""
     RECORD_SEPARATOR = u"\x1e"
     MAX_RECORD_SIZE = 1024 * 1024
     MAX_REQUEST_SIZE = 4 * 1024 * 1024

autopush/main.py

@@ -127,7 +127,8 @@ def add_shared_args(parser):
     parser.add_argument('-e', '--endpoint_port', help="HTTP Endpoint Port",
                         type=int, default=8082, env_var="ENDPOINT_PORT")
     parser.add_argument('--human_logs', help="Enable human readable logs",
-                        action="store_true", default=False)
+                        action="store_true", default=False,
+                        env_var="HUMAN_LOGS")
     parser.add_argument('--no_aws', help="Skip AWS meta information checks",
                         action="store_true", default=False)
     # No ENV because this is for humans

boto-compose.cfg

@@ -0,0 +1,9 @@
+[Credentials]
+aws_access_key_id =
+aws_secret_access_key =
+
+[Boto]
+is_secure = False
+https_validate_certificates = False
+proxy_port = 8000
+proxy = dynamodb

docker-compose.yml

@@ -0,0 +1,34 @@
+version: '2'
+services:
+  autopush:
+    image: bbangert/autopush
+    hostname: autopush
+    environment:
+      - LOCAL_HOSTNAME=localhost
+      - ROUTER_HOSTNAME=autopush
+      - HUMAN_LOGS=true
+      - CRYPTO_KEY=${CRYPTO_KEY}
+    ports:
+      - "8080:8080"
+      - "8081:8081"
+    links:
+      - dynamodb
+    volumes:
+      - ./boto-compose.cfg:/etc/boto.cfg:ro
+  autoendpoint:
+    image: bbangert/autopush
+    hostname: autoendpoint
+    command: autoendpoint
+    environment:
+      - HUMAN_LOGS=true
+      - CRYPTO_KEY=${CRYPTO_KEY}
+    ports:
+      - "8082:8082"
+    links:
+      - dynamodb
+    volumes:
+      - ./boto-compose.cfg:/etc/boto.cfg:ro
+  dynamodb:
+    image: peopleperhour/dynamodb
+    expose:
+      - "8000"

docs/api/logging.rst

@@ -5,12 +5,14 @@
 
 .. automodule:: autopush.logging
 
-.. autoclass:: EliotObserver
+.. autoclass:: PushLogger
     :members:
     :special-members: __init__, __call__
     :private-members:
     :member-order: bysource
 
-.. autofunction:: stdout
-
-.. autofunction:: setup_logging
+.. autoclass:: FirehoseProcessor
+    :members:
+    :special-members: __init__, __call__
+    :private-members:
+    :member-order: bysource

docs/architecture.rst

@@ -4,9 +4,34 @@
 Architecture
 ============
 
-Endpoint nodes handle all notification PUT requests, looking up in DynamoDB to
-see what Push server the UAID is connected to. The Endpoint nodes then attempt
-delivery to the Push server.
+.. image:: http://mozilla-push-service.readthedocs.io/en/latest/assets/push_architecture.svg
+
+Overview
+========
+
+For Autopush, we will focus on the section in the above diagram in the
+*Autopush* square.
+
+Autopush consists of two types of server daemons:
+
+``autopush`` (connection node)
+
+    Run a connection node. These handle large amounts of user agents (Firefox)
+    using the Websocket protocol.
+
+``autoendpoint`` (endpoint node)
+
+    Run an endpoint node. These provide a :term:`WebPush` HTTP API for
+    :term:`Application Servers <AppServer>` to HTTP POST messages to endpoints.
+
+To have a running Push Service for Firefox, both of these server daemons must
+be running and communicating with the same DynamoDB tables. A local DynamoDB
+can be run or AWS DynamoDB.
+
+Endpoint nodes handle all :term:`Notification` POST requests, looking up in
+DynamoDB to see what Push server the UAID is connected to. The Endpoint nodes
+then attempt delivery to the appropriate connection node. If the UAID is not
+online, the message may be stored in DynamoDB in the appropriate message table.
 
 Push connection nodes accept websocket connections (this can easily be HTTP/2
 for WebPush), and deliver notifications to connected clients. They check
@@ -15,12 +40,101 @@ DynamoDB for missed notifications as necessary.
 There will be many more Push servers to handle the connection node, while more
 Endpoint nodes can be handled as needed for notification throughput.
 
-Table Rotation
-==============
+.. _cryptography:
+
+Cryptography
+============
+
+The HTTP endpoint URL's generated by the connection nodes contain encrypted
+information, the :term:`UAID` and :term:`Subscription` to send the message to.
+This means that they both must have the same ``CRYPTO_KEY`` supplied to each.
+
+See :meth:`~autopush.settings.AutopushSettings.make_endpoint` for the endpoint
+URL generator.
+
+If you are only running Autopush locally, you can skip to :ref:`running` as
+later topics in this document apply only to developing or production scale
+deployments of Autopush.
+
+DynamoDB Tables
+===============
+
+Autopush uses a single router table and multiple messages tables, one for
+each month of the year. On startup, Autopush will create the router table and
+a message table for the prior month and the current month of the year.
+
+For more information on DynamoDB tables, see
+http://docs.aws.amazon.com/amazondynamodb/latest/gettingstartedguide/Welcome.html
+
+Router Table Schema
+-------------------
+
+The router table stores metadata for a given :term:`UAID` as well as which
+month table should be used for clients with a ``router_type`` of ``webpush``.
+
+For :term:`Bridging`, additional bridge-specific data may be stored in the
+router record for a :term:`UAID`.
+
+============   ==============
+uaid           **partition key** - :term:`UAID`
+router_type    :term:`Router Type`
+node_id        Hostname of the connection node the client is connected to.
+connected_at   Precise time (in milliseconds) the client connected to the node.
+last_connect   **global secondary index** - year-month-hour that the client has
+               last connected.
+curmonth       Message table name to use for storing :term:`WebPush` messages.
+============   ==============
+
+Autopush uses an optimistic deletion policy for ``node_id`` to avoid delete
+calls when not needed. During a delivery attempt, the endpoint will check the
+``node_id`` for the corresponding :term:`UAID`. If the client is not connected, it will
+clear the ``node_id`` record for that :term:`UAID` in the router table.
+
+If an endpoint node discovers during a delivery attempt that
+the ``node_id`` on record does not have the client connected, it will clear the
+``node_id`` record for that :term:`UAID` in the router table.
+
+The ``last_connect`` has a secondary global index on it to allow for maintenance
+scripts to locate and purge stale client records and messages.
+
+
+Clients with a ``router_type`` of ``webpush`` drain stored messages from the
+message table named ``curmonth`` after completing their initial handshake. If the
+``curmonth`` entry is not the current month then it updates it to store new
+messages in the latest message table after stored message retrieval.
+
+Message Table Schema
+--------------------
+
+The message table stores messages for users while they're offline or unable to
+get immediate message delivery.
+
+=============   ==============
+uaid            **partition key** - :term:`UAID`
+chidmessageid   **sort key** - :term:`CHID` + :term:`Message-ID`.
+chids           Set of :term:`CHID` that are valid for a given user. This
+                entry is only present in the item when ``chidmessageid`` is a space.
+data            Payload of the message, provided in the Notification body.
+headers         HTTP headers for the Notification.
+ttl             Time-To-Live for the Notification.
+timestamp       Time (in seconds) that the message was saved.
+updateid        UUID generated when the message is stored to track if the message
+                is updated between a client reading it and attempting to delete
+                it.
+=============   ==============
+
+The subscribed channels are stored as ``chids`` in a record stored with a blank
+space set for ``chidmessageid``. Before storing or delivering a :term:`Notification`
+a lookup is done against these ``chids``.
+
+.. _table-rotation:
+
+Message Table Rotation
+----------------------
 
 To avoid costly table scans, autopush uses a rotating message and router table.
-Clients that haven't connected in 30-60 days will have heir router and message
-table entries will dropped and need to re-register.
+Clients that haven't connected in 30-60 days will have their router and message
+table entries dropped and need to re-register.
 
 Tables are post-fixed with the year/month they are meant for, ie:
 
@@ -31,21 +145,25 @@ separate process in advance of the month switch-over as autopush nodes will
 assume the tables already exist. Scripts are provided that can be run weekly to
 ensure all necessary tables are present, and tables old enough are dropped.
 
+.. seealso::
+
+    Table maintenance script: https://github.com/mozilla-services/autopush/blob/master/maintenance.py
+
 Within a few days of the new month, the load on the prior months table will fall
 as clients transition to the new table. The read/write units on the prior
 month may then be lowered.
 
-Message Table
--------------
+Message Table Interaction Rules
+-------------------------------
 
 Due to the complexity of having notifications spread across two tables, several
 rules are used to avoid losing messages during the month transition.
 
 The logic for connection nodes is more complex, since only the connection node
 knows when the client connects, and how many messages it has read through.
 
-A new field will be added to the router table to indicate the last month the
-client has read connections through. This is independent of the last_connected
+The router table uses the ``curmonth`` field to indicate the last month the
+client has read notifications through. This is independent of the last_connect
 since it is possible for a client to connect, fail to read its notifications,
 then reconnect. This field is updated for a new month when the client connects
 **after** it has ack'd all the notifications out of the last month.

docs/glossary.rst

@@ -25,12 +25,31 @@ Glossary
 
     Message-ID
        A unique message ID. Each message for a given subscription is given a
-       unique identifier.
+       unique identifier that is returned to the :term:`AppServer` in the
+       ``Location`` header.
+
+    Notification
+       A message sent to an endpoint node intended for delivery to a HTTP
+       endpoint. Autopush stores these in the message tables.
+
+    Router Type
+       Every :term:`UAID` that connects has a router type. This indicates the
+       type of routing to use when dispatching notifications. For most clients, this
+       value will be ``webpush``. Older Firefox OS clients use ``simplepush``
+       and clients using :term:`Bridging` it will be one of ``gcm``, ``fcm``,
+       or ``apns``.
 
     Subscription
        A unique route between an :term:`AppServer` and the Application. May
        also be referred to as a :term:`Channel`
 
     UAID
        The Push User Agent Regisration ID. Push assigns each remote recipient
-       a unique indentifier.
+       (Firefox client) a unique identifier. These may occasionally be reset
+       by the Push Service or the client.
+
+    WebPush
+       An IETF standard for communication between Push Services, the clients,
+       and application servers.
+
+       See: https://datatracker.ietf.org/doc/draft-ietf-webpush-protocol/

docs/index.rst

@@ -14,20 +14,51 @@ This is the third generation of Push server built in Mozilla Services, first
 to handle Push for FirefoxOS clients, then extended for push notifications for
 Firefox (via the `W3C Push spec <http://w3c.github.io/push-api/index.html>`_.)
 
-Reference Docs
+For an overview of the Mozilla Push Service and where autopush fits in, see
+the `Mozilla Push Service architecture diagram`_. This push service uses
+websockets to talk to Firefox, with a Push endpoint that implements the
+:term:`WebPush` standard for its :ref:`http` API.
+
+Autopush API's
 ==============
 
+For developers writing mobile applications in Mozilla, or web developers using
+Push on the web with Firefox.
+
+.. toctree::
+    :maxdepth: 2
+
+    http
+
+.. _running-autopush:
+
+Running Autopush
+================
+
+If you just want to run autopush, for testing Push locally with
+Firefox, or to deploy autopush to a production environment for Firefox.
+
+.. toctree::
+   :maxdepth: 2
+
+   architecture
+   running
+
+.. _developing:
+
+Developing Autopush
+===================
+
+For developers wishing to work with the latest autopush source code, it's
+recommended that you first familiarize yourself with
+:ref:`running Autopush <running-autopush>` before proceeding.
+
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 2
 
    install
    testing
-   running
    releasing
-   architecture
-   api
-   Changelog <https://github.com/mozilla-services/autopush/blob/master/CHANGELOG.md>
-   http
    style
 
 Source Code
@@ -36,6 +67,15 @@ Source Code
 All source code is available on `github under autopush
 <https://github.com/mozilla-services/autopush>`_.
 
+Changelog
+=========
+
+.. toctree::
+   :maxdepth: 2
+
+   Changelog <https://github.com/mozilla-services/autopush/blob/master/CHANGELOG.md>
+
+
 Bugs/Support
 ============
 
@@ -70,8 +110,8 @@ production
 * Websocket: wss://push.services.mozilla.com/
 * Endpoint: https://updates.push.services.mozilla.com/
 
-Indices and tables
-==================
+Reference
+=========
 
 * :ref:`genindex`
 * :ref:`modindex`
@@ -88,3 +128,4 @@ License
 ``autopush`` is offered under the Apache License 2.0.
 
 .. _\#push: irc://irc.mozilla.org/push
+.. _Mozilla Push Service architecture diagram: http://mozilla-push-service.readthedocs.io/en/latest/#architecture