From 4282aec1f59920e8074c73e123449689507fdefe Mon Sep 17 00:00:00 2001 From: Conor Schaefer Date: Tue, 4 May 2021 15:43:09 -0700 Subject: [PATCH] Removes Xenial-specific references from the docs Follow-up to https://github.com/freedomofpress/securedrop/pull/5911, in which we removed Xenial-specific codepaths from the code repo. Here, we do the same throughout the documentation. There are notable exceptions: * https://docs.securedrop.org/en/stable/v3_services.html * https://docs.securedrop.org/en/stable/upgrade/focal_migration.html Those URLs remain untouched by these changes. We can follow up soon with removal for those, too, but it seems helpful to maintain a bit longer. --- docs/admin.rst | 9 -- docs/development/contributing.rst | 4 +- docs/development/dockerbuildmaint.rst | 3 +- docs/development/qubes_staging.rst | 109 +++++++----------- docs/development/release_management.rst | 2 +- docs/development/setup_development.rst | 8 +- .../testing_continuous_integration.rst | 67 ++++++----- docs/development/upgrade_testing.rst | 5 - docs/development/virtual_environments.rst | 18 ++- 9 files changed, 91 insertions(+), 134 deletions(-) diff --git a/docs/admin.rst b/docs/admin.rst index fad0bd3f1..af0cfd51d 100644 --- a/docs/admin.rst +++ b/docs/admin.rst @@ -437,15 +437,6 @@ VERSION_CODENAME is "focal" sudo unattended-upgrades - -VERSION_CODENAME is "xenial" -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: sh - - sudo cron-apt -i -s - - .. note:: Depending on the nature of the update (e.g., if the ``tor`` package is diff --git a/docs/development/contributing.rst b/docs/development/contributing.rst index a47b98538..4a0240c2a 100644 --- a/docs/development/contributing.rst +++ b/docs/development/contributing.rst @@ -154,8 +154,8 @@ are covered in :doc:`detailed documentation `. If you are a `Debian developer `__ you can help improve packaging and the release process: -* `Building SecureDrop application and OSSEC packages `__ and `pending bugs and tasks `__ -* Building `grsecurity kernels `__ and `pending bugs and tasks `__ +* `Building SecureDrop application and OSSEC packages `__ and `pending bugs and tasks `__ +* Building `grsecurity kernels `__ and `pending bugs and tasks `__ diff --git a/docs/development/dockerbuildmaint.rst b/docs/development/dockerbuildmaint.rst index f8cbc338f..d4f3c4650 100644 --- a/docs/development/dockerbuildmaint.rst +++ b/docs/development/dockerbuildmaint.rst @@ -5,8 +5,7 @@ Build container We use a Docker build container to build our debian packages for SecureDrop (via ``make build-debs`` in the ``securedrop`` Github repository root directory). We keep images of this our container in a Docker repository at **quay.io/freedomofpress**. The images are organized by Ubuntu release -version. For instance, you can find the images for Xenial at -**quay.io/freedomofpress/sd-docker-builder-xenial** and, for Focal, at +version. For instance, you can find the images for Focal at **quay.io/freedomofpress/sd-docker-builder-focal**. Maintaining images of our build container for each release is our way of recording the exact version diff --git a/docs/development/qubes_staging.rst b/docs/development/qubes_staging.rst index 5e28fe817..b07883b6f 100644 --- a/docs/development/qubes_staging.rst +++ b/docs/development/qubes_staging.rst @@ -1,9 +1,9 @@ Virtual Environments: Using Qubes ================================= -SecureDrop currently uses Ubuntu Xenial as its server OS, and Focal support is under -development. The instructions below cover setting up a SecureDrop staging environment -using either Xenial or Focal under Qubes. +SecureDrop currently uses Ubuntu Focal as its server OS. +The instructions below cover setting up a SecureDrop staging environment +using Focal under Qubes. It is assumed that you have an up-to-date Qubes installation on a compatible laptop, with at least 16GB RAM and 60GB free disk space. The SecureDrop server VMs @@ -12,21 +12,18 @@ accurately for Tor to start and hidden services to be available. Overview -------- -.. note:: Throughout the following instructions, ``$SERVER_OS`` will refer to your choice - of either ``xenial`` or ``focal``. - Follow the the Qubes platform instructions in :doc:`setup_development` to create a Debian 10 ``sd-dev`` Standalone VM. Once done, we'll create three new Standalone (HVM) Qubes VMs for use with staging: -- ``sd-staging-base-$SERVER_OS``, a base VM for cloning reusable staging VMs -- ``sd-staging-app-base-$SERVER_OS``, a base VM for the *SecureDrop Application Server* -- ``sd-staging-mon-base-$SERVER_OS``, a base VM for the *SecureDrop Monitor Server* +- ``sd-staging-base-focal``, a base VM for cloning reusable staging VMs +- ``sd-staging-app-base-focal``, a base VM for the *SecureDrop Application Server* +- ``sd-staging-mon-base-focal``, a base VM for the *SecureDrop Monitor Server* Download Ubuntu server ISO ---------------------------- -On ``sd-dev``, download the latest Ubuntu server ISO for either Xenial or Focal, +On ``sd-dev``, download the latest Ubuntu server ISO for Focal, along with corresponding checksum and signature files. See the :ref:`hardware installation docs ` for detailed instructions. If you opt for the command line instructions, omit @@ -42,11 +39,11 @@ In ``dom0``, do the following: .. code:: sh - qvm-create sd-staging-base-$SERVER_OS --class StandaloneVM --property virt_mode=hvm --label green - qvm-volume extend sd-staging-base-$SERVER_OS:root 20g - qvm-prefs sd-staging-base-$SERVER_OS memory 2000 - qvm-prefs sd-staging-base-$SERVER_OS maxmem 2000 - qvm-prefs sd-staging-base-$SERVER_OS kernel '' + qvm-create sd-staging-base-focal --class StandaloneVM --property virt_mode=hvm --label green + qvm-volume extend sd-staging-base-focal:root 20g + qvm-prefs sd-staging-base-focal memory 2000 + qvm-prefs sd-staging-base-focal maxmem 2000 + qvm-prefs sd-staging-base-focal kernel '' The commands above will create a new StandaloneVM, expand the storage space and memory available to it, as well as disable the integrated kernel support. @@ -59,7 +56,7 @@ In ``dom0``: .. code:: sh - qvm-start sd-staging-base-$SERVER_OS --cdrom=sd-dev:$ISO_PATH + qvm-start sd-staging-base-focal --cdrom=sd-dev:$ISO_PATH where ``ISO_PATH`` is the full path to the Ubuntu ISO previously downloaded on ``sd-dev``. @@ -68,11 +65,11 @@ Next, choose **Install Ubuntu**. For the most part, the install process matches the :ref:`hardware install flow `, with a few exceptions: - - Server IP address: use value returned by ``qvm-prefs sd-staging-base-$SERVER_OS ip``, + - Server IP address: use value returned by ``qvm-prefs sd-staging-base-focal ip``, with ``/24`` netmask suffix - - Gateway: use value returned by ``qvm-prefs sd-staging-base-$SERVER_OS visible_gateway`` + - Gateway: use value returned by ``qvm-prefs sd-staging-base-focal visible_gateway`` - For DNS, use Qubes's DNS servers: ``10.139.1.1`` and ``10.139.1.2``. - - Hostname: ``sd-staging-base-$SERVER_OS`` + - Hostname: ``sd-staging-base-focal`` - Domain name should be left blank Make sure to configure LVM and use **Virtual disk 1 (xvda 20.0GB Xen Virtual Block device)** @@ -86,7 +83,7 @@ Once installation is done, let the machine shut down and then restart it with .. code:: sh - qvm-start sd-staging-base-$SERVER_OS + qvm-start sd-staging-base-focal in ``dom0``. You should get a login prompt. @@ -94,7 +91,7 @@ Initial VM configuration ------------------------ Before cloning this machine, we'll update software to reduce provisioning time -on the staging VMs. In the new ``sd-staging-base-$SERVER_OS`` VM's console, do: +on the staging VMs. In the new ``sd-staging-base-focal`` VM's console, do: .. code:: sh @@ -122,7 +119,7 @@ to GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0" -When initial configuration is done, run ``qvm-shutdown sd-staging-base-$SERVER_OS`` to shut it down. +When initial configuration is done, run ``qvm-shutdown sd-staging-base-focal`` to shut it down. Clone VMs --------- @@ -134,19 +131,19 @@ documented below. Run the following in ``dom0``: .. code:: sh - qvm-clone sd-staging-base-$SERVER_OS sd-staging-app-base-$SERVER_OS - qvm-clone sd-staging-base-$SERVER_OS sd-staging-mon-base-$SERVER_OS - qvm-prefs sd-staging-app-base-$SERVER_OS ip 10.137.0.50 - qvm-prefs sd-staging-mon-base-$SERVER_OS ip 10.137.0.51 - qvm-tags sd-staging-app-base-$SERVER_OS add created-by-sd-dev - qvm-tags sd-staging-mon-base-$SERVER_OS add created-by-sd-dev + qvm-clone sd-staging-base-focal sd-staging-app-base-focal + qvm-clone sd-staging-base-focal sd-staging-mon-base-focal + qvm-prefs sd-staging-app-base-focal ip 10.137.0.50 + qvm-prefs sd-staging-mon-base-focal ip 10.137.0.51 + qvm-tags sd-staging-app-base-focal add created-by-sd-dev + qvm-tags sd-staging-mon-base-focal add created-by-sd-dev Now start both new VMs: .. code:: sh - qvm-start sd-staging-app-base-$SERVER_OS - qvm-start sd-staging-mon-base-$SERVER_OS + qvm-start sd-staging-app-base-focal + qvm-start sd-staging-mon-base-focal On the consoles which eventually appear, you should be able to log in with ``sdadmin/securedrop``. @@ -154,11 +151,8 @@ On the consoles which eventually appear, you should be able to log in with Configure cloned VMs ~~~~~~~~~~~~~~~~~~~~ -We'll need to fix each machine's idea of its own IP. The config location differs -on your OS choice: - -- **Xenial:** In the console for each machine, edit ``/etc/network/interfaces`` to update the ``address`` line with the machine's IP. -- **Focal:** In the console for each machine, edit ``/etc/netplan/00-installer-config.yaml`` to update the ``addresses`` entry with the machine's IP. +We'll need to fix each machine's idea of its own IP. In the console for each machine, +edit ``/etc/netplan/00-installer-config.yaml`` to update the ``addresses`` entry with the machine's IP. Edit ``/etc/hosts`` on each host to include the hostname and IP for itself. Use ``app-staging`` and ``mon-staging`` as appropriate. @@ -245,8 +239,9 @@ to set up the development environment. Once finished, build the Debian packages for installation on the staging VMs: -- **Xenial:** use the command ``make build-debs`` -- **Focal:** use the command ``make build-debs-focal`` +.. code:: sh + + make build-debs-focal Managing Qubes RPC for Admin API capability ------------------------------------------- @@ -293,25 +288,26 @@ Creating staging instance After creating the StandaloneVMs as described above: * ``sd-dev`` -* ``sd-staging-base-$SERVER_OS`` -* ``sd-staging-app-base-$SERVER_OS`` -* ``sd-staging-mon-base-$SERVER_OS`` +* ``sd-staging-base-focal`` +* ``sd-staging-app-base-focal`` +* ``sd-staging-mon-base-focal`` And after building the SecureDrop .debs, we can finally provision the staging environment: -- **Xenial:** run the command ``make staging`` -- **Focal:** run the command ``make staging-focal`` +.. code:: sh + + make staging -The commands invoke the appropriate Molecule scenario for your choice of ``$SERVER_OS``. +The commands invoke the appropriate Molecule scenario for your choice of ``focal``. You can also run constituent Molecule actions directly, rather than using the Makefile target: .. code:: sh - molecule create -s qubes-staging-$SERVER_OS - molecule converge -s qubes-staging-$SERVER_OS - molecule test -s qubes-staging-$SERVER_OS + molecule create -s qubes-staging-focal + molecule converge -s qubes-staging-focal + molecule test -s qubes-staging-focal That's it. You should now have a running, configured SecureDrop staging instance running on your Qubes machine. For day-to-day operation, you should run @@ -320,7 +316,7 @@ to provision staging VMs on-demand. To remove the staging instance, use the Mole .. code:: sh - molecule destroy -s qubes-staging-$SERVER_OS + molecule destroy -s qubes-staging-focal Accessing the Journalist Interface (Staging) in Whonix-based VMs ---------------------------------------------------------------- @@ -383,22 +379,3 @@ At this point, you should be able to access the *Journalist Interface* Note that you will have to replace the ``app-journalist.auth_private`` file and reload Tor on the Whonix gateway every time you rebuild the staging environment. - -Switching between Xenial and Focal ----------------------------------- - -Both environments may be set up on your Qubes workstation, but they cannot be run -simultaneously. To switch between them: - -- Use the appropriate ``molecule destroy`` command to bring down the active environment. -- Remove SSH known host entries for the servers with the commands: - - .. code:: sh - - ssh-keygen -f "/home/user/.ssh/known_hosts" -R "10.137.0.50" - ssh-keygen -f "/home/user/.ssh/known_hosts" -R "10.137.0.51" - - -- Build environment-specific packages first if necessary with ``make build-debs`` - or ``make build-debs-focal``. -- Run ``make staging`` or ``make staging-focal`` as appropriate. diff --git a/docs/development/release_management.rst b/docs/development/release_management.rst index 233f07e3e..0205c784d 100644 --- a/docs/development/release_management.rst +++ b/docs/development/release_management.rst @@ -255,7 +255,7 @@ Release Process #. In a clone of the private `securedrop-debian-packages-lfs `_ repository, create a branch from ``main`` called ``release``. -#. In your local branch, commit the built packages to the ``core/xenial`` +#. In your local branch, commit the built packages to the ``core/focal`` directory. * If the release includes a Tor update, make sure to include the diff --git a/docs/development/setup_development.rst b/docs/development/setup_development.rst index 5a646bbdd..8293b0ab7 100644 --- a/docs/development/setup_development.rst +++ b/docs/development/setup_development.rst @@ -195,7 +195,7 @@ Ansible on your development workstation. Ubuntu or Debian GNU/Linux ~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. note:: Tested on: Ubuntu 16.04 and Debian GNU/Linux stretch +.. note:: Tested on: Debian GNU/Linux 10 Buster .. code:: sh @@ -203,9 +203,9 @@ Ubuntu or Debian GNU/Linux sudo apt-get install -y build-essential libssl-dev libffi-dev python3-dev \ dpkg-dev git linux-headers-$(uname -r) virtualbox -We recommend using the latest stable version of Vagrant, ``1.8.5`` at the time -of this writing, which might be newer than what is in your distro's package -repositories. Older versions of Vagrant has been known to cause problems +We recommend using the most recent version of Vagrant available in your distro's +package repositories. For Debian Stable, that's ``2.2.3`` at the time +of this writing. Older versions of Vagrant has been known to cause problems (`GitHub #932`_, `GitHub #1381`_). If ``apt-cache policy vagrant`` says your candidate version is not at least 1.8.5, you should download the current version from the `Vagrant Downloads page`_ and then install it. diff --git a/docs/development/testing_continuous_integration.rst b/docs/development/testing_continuous_integration.rst index 235121727..b788b0dc3 100644 --- a/docs/development/testing_continuous_integration.rst +++ b/docs/development/testing_continuous_integration.rst @@ -89,7 +89,6 @@ Run ``make help`` to see the full list of CI commands in the Makefile: Makefile for developing and testing SecureDrop. Subcommands: ci-go Creates, provisions, tests, and destroys GCE host for testing staging environment. - ci-go-xenial Creates, provisions, tests, and destroys GCE host for testing staging environment under xenial. ci-lint Runs linting in linting container. ci-teardown Destroys GCE host for testing staging environment. @@ -117,48 +116,48 @@ of a commit to apply to a branch in order disable the deletion for the Focal sta .. code:: Diff - diff --git a/.circleci/config.yml b/.circleci/config.yml - index 4a9b0bd4c..d9aea01b8 100644 - --- a/.circleci/config.yml - +++ b/.circleci/config.yml - @@ -354,13 +354,6 @@ jobs: - BASE_OS=focal make ci-go - no_output_timeout: 35m - - - - run: - - name: Ensure environment torn down - - # Always report true, since env should will destroyed already - - # if all tests passed. - - command: make ci-teardown || true - - when: always - - - - store_test_results: - path: ~/sd/junit - - diff --git a/devops/gce-nested/ci-go.sh b/devops/gce-nested/ci-go.sh - index 850324ecc..776120df4 100755 - --- a/devops/gce-nested/ci-go.sh - +++ b/devops/gce-nested/ci-go.sh - @@ -16,4 +16,3 @@ export BASE_OS="${BASE_OS:-xenial}" + diff --git a/.circleci/config.yml b/.circleci/config.yml + index 4d61769f1..af74672bc 100644 + --- a/.circleci/config.yml + +++ b/.circleci/config.yml + @@ -251,13 +251,6 @@ jobs: + make ci-go + no_output_timeout: 35m + + - - run: + - name: Ensure environment torn down + - # Always report true, since env should will destroyed already + - # if all tests passed. + - command: make ci-teardown || true + - when: always + - + - store_test_results: + path: ~/sd/junit + + diff --git a/devops/gce-nested/ci-go.sh b/devops/gce-nested/ci-go.sh + index ff80aa107..65bbcd7b9 100755 + --- a/devops/gce-nested/ci-go.sh + +++ b/devops/gce-nested/ci-go.sh + @@ -16,4 +16,3 @@ export BASE_OS="${BASE_OS:-focal}" ./devops/gce-nested/gce-start.sh ./devops/gce-nested/gce-runner.sh - -./devops/gce-nested/gce-stop.sh - diff --git a/devops/scripts/create-staging-env b/devops/scripts/create-staging-env - index 3b9a2c7f8..df2ccfe3d 100755 - --- a/devops/scripts/create-staging-env - +++ b/devops/scripts/create-staging-env - @@ -33,7 +33,7 @@ printf "Creating staging environment via '%s'...\\n" "${securedrop_staging_scena + -./devops/gce-nested/gce-stop.sh + diff --git a/devops/scripts/create-staging-env b/devops/scripts/create-staging-env + index 8b296be94..df8a4d674 100755 + --- a/devops/scripts/create-staging-env + +++ b/devops/scripts/create-staging-env + @@ -32,7 +32,7 @@ printf "Creating staging environment via '%s'...\\n" "${securedrop_staging_scena virtualenv_bootstrap # Are we in CI? Then lets do full testing post install! if [ "$USER" = "sdci" ]; then - - molecule test -s "${securedrop_staging_scenario}" - + molecule test --destroy=never -s "${securedrop_staging_scenario}" + - molecule test -s "${securedrop_staging_scenario}" + + molecule test --destroy=never -s "${securedrop_staging_scenario}" else - molecule "${MOLECULE_ACTION:-converge}" -s "${securedrop_staging_scenario}" "${EXTRA_ANSIBLE_ARGS[@]}" + molecule "${MOLECULE_ACTION:-converge}" -s "${securedrop_staging_scenario}" "${EXTRA_ANSIBLE_ARGS[@]}" fi -Once that commit is pushed, run the appropriate ``staging-test-with-rebase`` job +Once that commit is pushed, run the ``staging-test-with-rebase`` job with ssh using with CircleCI. Once logged into that container, you can ssh into the Google Compute host: diff --git a/docs/development/upgrade_testing.rst b/docs/development/upgrade_testing.rst index 8f34324de..3795b70e7 100644 --- a/docs/development/upgrade_testing.rst +++ b/docs/development/upgrade_testing.rst @@ -29,11 +29,6 @@ https://apt-test.freedom.press/. Both options are described below. Upgrade testing using locally-built packages -------------------------------------------- -.. note:: - As of ``0.12.1``, the default platform for upgrade testing - boxes is Ubuntu Xenial 16.04. We no longer support upgrade boxes - based on Ubuntu Trusty 14.04. - First, build the app code packages and create the environment: .. code:: sh diff --git a/docs/development/virtual_environments.rst b/docs/development/virtual_environments.rst index 6a7f690e5..7363d146b 100644 --- a/docs/development/virtual_environments.rst +++ b/docs/development/virtual_environments.rst @@ -42,16 +42,12 @@ Debian packages on the staging machines: make build-debs make staging - # Use the proper backend for your developer environment: - molecule login -s virtualbox-staging-xenial -h app-staging - # or: - molecule login -s libvirt-staging-xenial -h app-staging + molecule login -s libvirt-staging-focal -h app-staging sudo -u www-data bash cd /var/www/securedrop ./manage.py add-admin - pytest -v tests/ -To rebuild the local packages for the app code and update on Xenial staging: +To rebuild the local packages for the app code and update the staging VMs: .. code:: sh @@ -139,7 +135,7 @@ alert emails. Direct SSH access is available for staging hosts, so you can use ``molecule login -s -h app-staging``, where ```` -is either ``virtualbox-staging-xenial`` or ``libvirt-staging-xenial``, depending +is either ``libvirt-staging-focal`` or ``qubes-staging-focal``, depending on your environment. By default, the staging environments are created with an empty submissions database. If you want to set up a staging environment with a preexisting submissions database, you can do so using a SecureDrop backup file as follows: @@ -189,7 +185,7 @@ Switching to the Vagrant libvirt provider Make sure you've already installed Vagrant, as described in the :ref:`multi-machine setup docs `. -Ubuntu 16.04 setup +Ubuntu 20.04 setup ^^^^^^^^^^^^^^^^^^ Install libvirt and QEMU: @@ -276,12 +272,12 @@ Set the default Vagrant provider to ``libvirt``: Convert Vagrant boxes to libvirt ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Convert the VirtualBox images for Xenial from ``virtualbox`` to ``libvirt`` format: +Convert the VirtualBox images for Focal from ``virtualbox`` to ``libvirt`` format: .. code:: sh - vagrant box add --provider virtualbox bento/ubuntu-16.04 - vagrant mutate bento/ubuntu-16.04 libvirt + vagrant box add --provider virtualbox bento/ubuntu-20.04 + vagrant mutate bento/ubuntu-20.04 libvirt You can now use the libvirt-backed VM images to develop against the SecureDrop multi-machine environment.