Updates developer docs for upgrade testing targets

Makefile

@@ -163,20 +163,25 @@ staging-xenial: ## Creates local staging VMs based on Xenial, autodetecting plat
 clean: ## DANGER! Purges all site-specific info and developer files from project.
 	@./devops/clean
 
-.PHONY: upgrade_start
-upgrade_start: ## Boot up an upgrade test base environment using libvirt
+.PHONY: upgrade-start
+upgrade-start: ## Boot up an upgrade test base environment using libvirt
 	@SD_UPGRADE_BASE=$(STABLE_VER) molecule converge -s upgrade
 
-.PHONY: upgrade_destroy
-upgrade_destroy: ## Destroy up an upgrade test base environment
+.PHONY: upgrade-start-qa
+upgrade-start-qa: ## Boot up an upgrade test base env using libvirt in remote apt mode
+	@SD_UPGRADE_BASE=$(STABLE_VER) QA_APTTEST=yes molecule converge -s upgrade
+
+.PHONY: upgrade-destroy
+upgrade-destroy: ## Destroy up an upgrade test base environment
 	@SD_UPGRADE_BASE=$(STABLE_VER) molecule destroy -s upgrade
 
-.PHONY: upgrade_test_local
-upgrade_test_local: ## Once an upgrade environment is running, force upgrade apt packages (local pkgs)
+.PHONY: upgrade-test-local
+upgrade-test-local: ## Once an upgrade environment is running, force upgrade apt packages (local pkgs)
 	@molecule side-effect -s upgrade
 
-.PHONY: upgrade_test_qa
-upgrade_test_qa: ## Once an upgrade environment is running, force upgrade apt packages (from qa server)
+.PHONY: upgrade-test-qa
+upgrade-test-qa: ## Once an upgrade environment is running, force upgrade apt packages (from qa server)
+	@QA_APTTEST=yes molecule converge -s upgrade -- --diff -t apt
 	@QA_APTTEST=yes molecule side-effect -s upgrade
 
 # Explaination of the below shell command should it ever break.

docs/development/upgrade_testing.rst

@@ -6,23 +6,23 @@ Upgrade Testing using Molecule
 The SecureDrop project includes Molecule scenarios for developing and testing against
 multi-server configurations, including a scenario to simulate the process of upgrading an
 existing system. This document explains how to work with this scenario to test
-features that make potentially release-breaking changes such as database 
+features that make potentially release-breaking changes such as database
 schema updates.
 
-The Molecule upgrade scenario sets up a predefined staging Securedrop virtual 
+The Molecule upgrade scenario sets up a predefined staging Securedrop virtual
 environment using Vagrant boxes built with the latest application release.
-It also creates a virtualized APT repository, and modifies 
-the SecureDrop environment to use this APT repository instead of the FPF main 
-repo at https://apt.freedom.press/. 
+It also creates a virtualized APT repository, and modifies
+the SecureDrop environment to use this APT repository instead of the FPF main
+repo at https://apt.freedom.press/.
 
-You can use this scenario to test the upgrade process, using using either 
-locally-built .debs or packages from the FPF test repo at 
+You can use this scenario to test the upgrade process, using using either
+locally-built .debs or packages from the FPF test repo at
 https://apt-test.freedom.press/. Both options are described below.
-  
-.. note:: The upgrade scenario uses QEMU/KVM via Vagrant's libvirt provider, in 
-   place of the  default Virtualbox provider. If you haven't already done so, 
-   you'll need to set up the libvirt provider before proceeding. For 
-   more information, see :ref:`libvirt_provider`. 
+
+.. note:: The upgrade scenario uses QEMU/KVM via Vagrant's libvirt provider, in
+   place of the  default Virtualbox provider. If you haven't already done so,
+   you'll need to set up the libvirt provider before proceeding. For
+   more information, see :ref:`libvirt_provider`.
 
 .. _upgrade_testing_local:
 
@@ -34,92 +34,99 @@ First, build the app code packages and create the environment:
 .. code:: sh
 
  make build-debs
- molecule converge -s upgrade
+ make upgrade-start
 
 The playbook will return the source interface Onion address. You can use this to
-check the application version displayed in the source interface footer. 
-Alternatively, you can log into the Application Server VM and check the deployed 
+check the application version displayed in the source interface footer.
+Alternatively, you can log into the *Application Server* VM and check the deployed
 package version directly:
 
 .. code:: sh
 
    molecule login -s upgrade -h app-staging
 
-From the Application Server:
+From the *Application Server*:
 
 .. code:: sh
 
    apt-cache-policy securedrop-config
 
 The installed package version should match the latest release version.
 
-To perform an upgrade using the virtualized APT repository, log out of the 
-Application Server and run the Molecule side-effect action:
+To perform an upgrade using the virtualized APT repository, log out of the
+*Application Server* and run the Molecule side-effect action:
 
 .. code:: sh
 
-   molecule side-effect -s upgrade
+   make upgrade-test-local
 
-This will upgrade the SecureDrop packages on the application and 
-monitor servers,
-using your locally-built packages and APT VM instead of the FPF main APT 
-repository. 
+This will upgrade the SecureDrop packages on the *Application* and
+*Monitor Servers*, using your locally-built packages and apt VM instead of the
+FPF production apt repository.
 
-You can verify that the application version has changed either by checking the 
-source interface's footer or directly on the Application Server as described 
-above. 
+You can verify that the application version has changed either by checking the
+source interface's footer or directly on the *Application Server* as described
+above.
 
 .. _upgrade_testing_apt:
 
 Upgrade testing using apt-test.freedom.press
 --------------------------------------------
 
-You can use the upgrade scenario to test upgrades using official release 
-candidate packages from the FPF test APT repository. First, 
+You can use the upgrade scenario to test upgrades using official release
+candidate packages from the FPF test APT repository. First,
 create the environment:
 
 .. code:: sh
 
-   make build-debs 
-   molecule converge -s upgrade
+   make upgrade-start-qa
 
-Then, log into the Application Server:
+Then, log into the *Application Server*:
 
 .. code:: sh
-   
+
    molecule login -s upgrade -h app-staging
 
-From the Application Server:
+From the *Application Server*:
 
 .. code:: sh
 
    sudo apt-get update
    apt-cache policy securedrop-config
 
-The installed package version should match the current release version, and the
-candidate version should match your locally-built version.
-
-Now, log out of the app server. To switch to the apt-test proxy:
+The installed package version should match the current release version.
+To install the latest packages from the apt-test proxy:
 
 .. code:: sh
 
-   QA_APTTEST=yes molecule converge -s upgrade -- --diff -t apt
+   make upgrade-test-qa
 
-Log back into the Application Server, and repeat the previous commands:
+Log back into the *Application Server*, and repeat the previous commands:
 
 .. code:: sh
 
    sudo apt-get update
-   apt-cache policy securedrop-config 
-
-This time, you should see multiple entries in the version table, corresponding 
-to the versions available on the FPF test APT repository. If a new release 
-candidate is available, you can use the molecule side-effect action from your 
-local terminal to perform an upgrade:
-
-.. code:: sh
-
-   molecule side-effect -s upgrade
+   apt-cache policy securedrop-config
 
-Navigate to the Source Interface URL again, and confirm you see the upgraded 
+Navigate to the Source Interface URL again, and confirm you see the upgraded
 version in the footer. Then proceed with testing the new version.
+
+Updating the base boxes used for upgrade testing
+------------------------------------------------
+
+When a new version of SecureDrop is released, we must create and upload
+new VM images, to enable testing against that base version in future upgrade
+testing. The procedure is as follows:
+
+1. ``git checkout <version>``
+2. ``make vagrant-package``
+3. ``mv molecule/vagrant_packager/build/app-staging{,_<version>}.box``
+4. ``mv molecule/vagrant_packager/build/mon-staging{,_<version>}.box``
+5. ``sha256sum molecule/vagrant_packager/build/*.box``
+6. Manually update ``molecule/vagrant_packager/box_files/*.json`` with new
+   version information, including URL and checksum.
+7. ``cd molecule/vagrant_packager && ./push.yml`` to upload to S3
+8. Commit the local changes to JSON files and open a PR.
+
+Subsequent invocations of ``make upgrade-start`` will pull the latest
+version of the box.

molecule/upgrade/playbook.yml

@@ -1,4 +1,30 @@
 ---
+- name: Confirm local debs present if required
+  hosts: localhost
+  gather_facts: no
+  vars:
+    molecule_dir: "{{ lookup('env', 'MOLECULE_SCENARIO_DIRECTORY') }}"
+  tasks:
+      # Before running apt-server logic, confirm we have local
+      # debs built.
+    - name: Establish QA strategy
+      set_fact:
+        QA_APTTEST: "{{ lookup('env','QA_APTTEST')|bool }}"
+
+    - name: Find local deb files
+      find:
+        paths: "{{ molecule_dir }}/../../build/"
+        patterns: "*.deb"
+      register: _upgrade_scenario_find_debs_result
+      when: not QA_APTTEST
+
+    - name: Ensure debs were found
+      assert:
+        that:
+          - "_upgrade_scenario_find_debs_result.files|length >= 8"
+        msg: "No local debs found, run 'make build-debs'"
+      when: not QA_APTTEST
+
 - name: Re-run original tor role
   hosts: securedrop
   max_fail_percentage: 0

molecule/vagrant_packager/box_files/app_metadata.json

@@ -34,6 +34,17 @@
           "checksum": "6fb8c12821b902b0905f967d7405c620443f4f4345b18d05b59005f798a08e50"
         }
       ]
+    },
+    {
+      "version": "0.11.0",
+      "providers": [
+        {
+          "name": "libvirt",
+          "url": "https://s3.amazonaws.com/securedrop-vagrant/app-staging_0.11.0.box",
+          "checksum_type": "sha256",
+          "checksum": "cf2c4c8ac89bd132d3f75ee77e2d97b68c562e1ee36aa059bb1d4e6b37499f62"
+        }
+      ]
     }
   ]
 }

molecule/vagrant_packager/box_files/mon_metadata.json

@@ -34,6 +34,17 @@
           "checksum": "9891c88aa7148129f2f91638d7dfed1e7815eb980bba1de8a9c075f14ae0ddeb"
         }
       ]
+    },
+    {
+      "version": "0.11.0",
+      "providers": [
+        {
+          "name": "libvirt",
+          "url": "https://s3.amazonaws.com/securedrop-vagrant/mon-staging_0.11.0.box",
+          "checksum_type": "sha256",
+          "checksum": "baba21e8799fe2093d902b332b45d7a8342adf019fa195382011fbdfa54cd1d5"
+        }
+      ]
     }
   ]
 }