Merge pull request #615 from zenhack/keystone-auth

Keystone auth

.travis.yml

@@ -28,8 +28,10 @@ env:
   - DB=postgres
 
 install:
-  - sh -e ci/install.sh
+  - sh -e ci/apache/install.sh
+  - sh -e ci/keystone/install.sh
 
 script:
   - sh -e ci/run_unit_tests.sh
-  - sh -e ci/run_integration_tests.sh
+  - sh -e ci/apache/run_integration_tests.sh
+  - sh -e ci/keystone/run_integration_tests.sh

INSTALL.rst

@@ -269,14 +269,25 @@ Authentication and Authorization
 --------------------------------
 
 HaaS includes a pluggable architecture for authentication and authorization.
-HaaS ships with two authentication backends. One uses HTTP basic auth, with
-usernames and passwords stored in the haas database. The other is a "null"
-backend, which does no authentication or authorization checks. This can be
-useful for testing and experimentation but *should not* be used in production.
-You must enable exactly one auth backend.
-
-Database Backend
-^^^^^^^^^^^^^^^^
+HaaS ships with a handful of authentication backends; you must enable exactly
+one of them. The list of incuded backends is:
+
+* A backend using HTTP basic auth, with usernames and passwords stored in the
+  HaaS database.
+* A backend which authenticates against Openstack's identity service. In order
+  to use this you must install the keystonemiddleware python package.
+  `keystone-auth.md <docs/keystone-auth.md>`_ Provides further information on
+  the use of tis backend.
+* The other is a "null" backend, which does no authentication or authorization
+  checks. This can be useful for testing and experimentation but *should not*
+  be used in production.
+
+It is also possible for third parties to supply authentication backends as
+extensions. If you wish to use such an extension, refer to the documentation
+for that extension.
+
+Database Backend (haas.ext.auth.database)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To enable the database backend, make sure the **[extensions]** section of
 ``haas.cfg`` contains::
@@ -293,14 +304,18 @@ within the directory containing the server's ``haas.cfg``::
 You can then create additional users via the HTTP API. You may want to
 subsequently delete the initial user; this can also be done via the API.
 
-Null Backend
-^^^^^^^^^^^^
+Keystone Backend (haas.ext.auth.keystone)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+See `The document describing this extension in detail. <docs/keystone-auth.md>`_
+
+Null Backend (haas.ext.auth.null)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To enable the null backend, make sure **[extensions]** contains::
 
   haas.ext.auth.null =
 
-
 Running the Server under Apache
 -------------------------------
 
@@ -334,7 +349,7 @@ allow <https://stackoverflow.com/questions/4390436/need-to-allow-encoded-slashes
 this due to security concerns. ``AllowEncodedSlashes On`` enables the passing
 of these arguments.
 
-**Note:** For apache to be able to pass the authentication headers to HaaS 
+**Note:** For apache to be able to pass the authentication headers to HaaS
 following directive will have to be turned on
 
 ``WSGIPassAuthorization On``

ci/install.sh → ci/apache/install.sh

@@ -2,7 +2,7 @@
 
 # Setup configuration
 cp ci/testsuite.cfg.$DB testsuite.cfg
-sudo cp ci/haas.cfg.apache.$DB /etc/haas.cfg
+sudo cp ci/apache/haas.cfg.$DB /etc/haas.cfg
 sudo chown travis:travis /etc/haas.cfg
 
 # Database Setup

ci/keystone/.gitignore

@@ -0,0 +1 @@
+/keystone

ci/keystone/install.sh

@@ -0,0 +1,8 @@
+#!/usr/bin/env sh
+set -ex
+pip install keystonemiddleware
+pip install python-keystoneclient
+# The exact commit we use here is somewhat arbitrary, but we want
+# something that (a) won't change out from under our feet, and (b)
+# works with our existing tests.
+keystone_commit=10.0.0.0b2 ./ci/keystone/keystone.sh setup

ci/keystone/keystone.sh

@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+#
+# Helper script for setting up and running keystone. Most of this is taken from:
+#
+#   http://docs.openstack.org/developer/keystone/developing.html
+
+# tunable variables:
+
+# So we can avoid repeatedly downloading the repo when developing this script:
+keystone_repo=${keystone_repo:-https://git.openstack.org/openstack/keystone}
+
+# git commit to use. Travis uses this to test against different releases of
+# keystone:
+keystone_commit=${keystone_commit:-master}
+
+# virtualenv executable; on some systems it may be named something else, e.g.
+# if we want to test against python 2 on a system for which python 3 is the
+# default, the executable will be virtualenv2:
+virtualenv_bin=${virtualenv_bin:-virtualenv}
+
+# -e: Stop on the first failing command.
+# -x: Print commands as they are executed.
+set -ex
+
+# If we're already in a virtualenv, deactivate it; we want to use one that just
+# has keystone dependencies. `|| true` prevents this from failing if we're not
+# in a venv.
+deactivate || true
+
+# Make sure we're in the directory containing this script; this way the user
+# can call it from anywhere.
+cd "$(dirname $0)"
+
+case "$1" in
+  setup)
+
+    git clone ${keystone_repo} keystone
+    cd keystone
+    git checkout ${keystone_commit}
+
+    ${virtualenv_bin} .venv
+    source .venv/bin/activate
+
+    # On some distros (e.g. Ubuntu 14.04), the installed version of pip is
+    # too old to parse some of the syntax used in keystone's requirements.txt.
+    # Make sure we have the latest:
+    pip install --upgrade pip
+
+    pip install -r requirements.txt
+    pip install .
+    pip install uwsgi # To actually run keystone; no webserver in the deps.
+
+    cp etc/keystone.conf.sample etc/keystone.conf
+
+    keystone-manage db_sync
+
+    # Populate the database with some sample data. First, make sure keystone is
+    # running:
+    ../keystone.sh run &
+    pid=$!
+    # Doing this after launching keystone will give it plenty of time to get
+    # started without adding any wasteful calls to sleep:
+    pip install python-openstackclient
+    source ../keystonerc # for $OS_PASSWORD
+    ADMIN_PASSWORD=s3cr3t ./tools/sample_data.sh
+
+    # In addition to the sample data from the keystone project's script above,
+    # we add an extra project and user for use in the tests:
+    openstack project create non-haas-project
+    openstack user create \
+      non-haas-user \
+      --password secret
+    openstack role add \
+      --project non-haas-project \
+      --user non-haas-user \
+      service
+
+    # stop the server:
+    kill $pid
+    wait
+
+    ;;
+  run)
+    cd keystone
+    source .venv/bin/activate
+    uwsgi \
+      --http 127.0.0.1:35357 \
+      --wsgi-file "$(which keystone-wsgi-admin)" \
+      --ini ../uwsgi.ini &
+    admin_pid=$!
+    uwsgi \
+      --http 127.0.0.1:5000 \
+      --wsgi-file "$(which keystone-wsgi-public)" \
+      --ini ../uwsgi.ini &
+    public_pid=$!
+    # If we're killed, propogate the signal to our children.
+    trap "kill $public_pid; kill $admin_pid" INT TERM
+    wait $public_pid
+    wait $admin_pid
+    ;;
+  *)
+    echo "Usage: $0 (setup|run)" >&2
+    exit 1
+    ;;
+esac

ci/keystone/keystonerc

@@ -0,0 +1,7 @@
+export OS_USERNAME=admin
+export OS_PASSWORD=s3cr3t
+export OS_PROJECT_NAME=admin
+export OS_USER_DOMAIN_ID=default
+export OS_PROJECT_DOMAIN_ID=default
+export OS_IDENTITY_API_VERSION=3
+export OS_AUTH_URL=http://localhost:5000/v3

ci/keystone/run_integration_tests.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env sh
+
+set -ex
+
+./ci/keystone/keystone.sh run &
+
+# Wait for curl to successfully connect to each of the ports keystone
+# is supposed to be listening on before continuing.
+for port in 5000 35357; do
+	while [ "$(curl http://127.0.0.1:$port; echo $?)" -ne 0 ]; do
+		sleep .2
+	done
+done
+
+py.test tests/integration/keystone.py

ci/keystone/uwsgi.ini

@@ -0,0 +1,19 @@
+[uwsgi]
+# misc. settings copied from devstack:
+#
+# https://github.com/openstack-dev/devstack/blob/e88c51cc1b0aa59abbae353f3fd3c2ef58e1602a/lib/keystone#L304-L342
+#
+# I (zenhack) did this after talking to stevemar in #openstack-keystone; some
+# intermittent issues magically solved themselves when adding these. My money is
+# on `add-header` as the important one, but some of the other stuff is still
+# useful sinces it mean the call to `kill` in keystone.sh actually gets the
+# worker processes too.
+#
+# This file is common to both the "public" and "admin" endpoints; the
+# per-endpoint options are passed on the command line in keystone.sh
+master = true
+die-on-term = true
+exit-on-reload = true
+enable-threads = true
+
+add-header = Connection: close

docs/extensions.rst

@@ -39,3 +39,14 @@ Extension-approved components currently include:
 * The migration framework; see ``migrations.md`` for an overview.
 
 See the docstrings for each component for details.
+
+Additionally, extensions may add wsgi middleware to the flask
+application from their ``setup`` function. For example:
+
+    app.wsgi_app = my_middleware(app.wsgi_app)
+
+Note that the order in which the ``setup`` functions are run is not
+defined. As such, if multiple extensions add wsgi middleware the
+order in which they are applied is also undefined. Using more than one
+such extension is discouraged. An ordering *may* be defined in the
+future.

docs/keystone-auth.md

@@ -0,0 +1,50 @@
+An authentication backend for Openstack's Keystone is maintained in this
+source tree as `haas.ext.auth.keystone`. This document describes its
+configuration and usage in detail.
+
+NOTE: The HaaS command line interface only supports the keystone v3 API.
+The server supports anything supported by [keystonemiddleware][1].
+
+# Usage
+
+Once HaaS has been configured to work with Keystone, an administrator
+must manually add Openstack projects to HaaS before they can access the
+HaaS API. The HaaS project names must correspond to the Openstack UUIDs.
+For example, an administrator may execute the command:
+
+    haas project_create 00de7c85e594473db7461cdf7367166a
+
+To grant the Openstack project with that UUID access to HaaS.
+
+Note that the plugin recognizes any user with an `admin` role on any
+project as a HaaS administrator, similar to the default policy for core
+Openstack projects.
+
+The HaaS command line interface will look for the same `OS_*`
+environment variables used by the Openstack command line tools; these
+may be set by a user to authenticate when using the CLI.
+
+A script to set these variables correctly can be downloaded from the
+Openstack web dashboard via "Access & Security."
+
+# Configuration
+
+As with any other extension, you must load the extension in `haas.cfg`:
+
+    [extensions]
+    haas.ext.auth.keystone =
+
+The backend must then be configured to talk to your keystone server.
+The keystone project maintains documentation on how to do this at:
+
+<http://docs.openstack.org/developer/keystonemiddleware/middlewarearchitecture.html>
+
+Configuring HaaS to talk to Keystone deviates in the following ways:
+
+* The paste configuration is not used; you can simply ignore the
+  sections that refer to paste.
+* The options that the Keystone documentation puts in the section
+  `[keystone_authtoken]` should instead be placed in the extension's
+  section in `haas.cfg`, i.e. `[haas.ext.auth.keystone]`.
+
+[1]: http://docs.openstack.org/developer/keystonemiddleware/