diff --git a/docs/development/setup_development.rst b/docs/development/setup_development.rst index 8293b0ab7..0ed9c02fd 100644 --- a/docs/development/setup_development.rst +++ b/docs/development/setup_development.rst @@ -8,12 +8,10 @@ Overview SecureDrop is a multi-machine design. To make development and testing easy, we provide a set of virtual environments, each tailored for a -specific type of development task. We use Vagrant, VirtualBox, and +specific type of development task. We use Vagrant, Molecule, and Docker and our Ansible playbooks can provision these environments on either virtual machines or physical hardware. -.. note:: SecureDrop is written in Python 3 only. - Quick Start ----------- @@ -188,7 +186,7 @@ Setting Up a Multi-Machine Environment .. note:: You do not need this step if you only plan to work on the web application or the documentation. -To get started, you will need to install Vagrant, VirtualBox, Docker, and +To get started, you will need to install Vagrant, Libvirt, Docker, and Ansible on your development workstation. @@ -201,7 +199,7 @@ Ubuntu or Debian GNU/Linux sudo apt-get update sudo apt-get install -y build-essential libssl-dev libffi-dev python3-dev \ - dpkg-dev git linux-headers-$(uname -r) virtualbox + dpkg-dev git linux-headers-$(uname -r) We recommend using the most recent version of Vagrant available in your distro's package repositories. For Debian Stable, that's ``2.2.3`` at the time @@ -227,9 +225,6 @@ from the `Vagrant Downloads page`_ and then install it. instructions in Vagrantfile that would enable vagrant-cachier are currently commented out. -VirtualBox should be at least version 5.x. See `GitHub #1381`_ for documentation -of incompatibility with the older VirtualBox 4.x release series. - Finally, install Ansible so it can be used with Vagrant to automatically provision VMs. We recommend installing Ansible from PyPi with ``pip`` to ensure you have the latest stable version. @@ -259,41 +254,8 @@ development-related tooling. Using `virtualenvwrapper macOS ~~~~~ -Install the dependencies for the development environment: - -#. Vagrant_ -#. VirtualBox_ -#. Ansible_ -#. rsync >= 3.1.0 - -If you use homebrew-cask_ to manage macOS apps, you can install Vagrant and -VirtualBox that way. As for Ansible, we strongly recommend installing it -in a virtual environment using ``virtualenvwrapper`` and ``pip``, so as not to -install the older version we use system-wide. You should create your virtualenv -using the Python 3 install on your system. If you are using a -different version, the path to ``virtualenvwrapper.sh`` will differ. Running -``pip show virtualenvwrapper`` should help you find it. - -.. code:: sh - - sudo easy_install pip # if you don't already have pip - sudo -H pip install -U virtualenvwrapper --ignore-installed six - source /usr/local/bin/virtualenvwrapper.sh - mkvirtualenv -p python3 securedrop - -.. note:: You'll want to add the command to source ``virtualenvwrapper.sh`` - to your ``~/.bashrc`` (or whatever your default shell configuration - file is) so that the command-line utilities ``virtualenvwrapper`` - provides are automatically available in the future. - -The version of rsync installed by default on macOS is extremely out-of-date, as is Apple's custom. We recommend using Homebrew_ to install a modern version (3.1.0 or greater): ``brew install rsync``. - -.. _Vagrant: https://www.vagrantup.com/downloads -.. _VirtualBox: https://www.virtualbox.org/wiki/Downloads -.. _Ansible: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html -.. _Homebrew: https://brew.sh/ -.. _homebrew-cask: https://sourabhbajaj.com/mac-setup/Vagrant/README.html - +Developers on macOS should use the Docker-based container environment. +We don't support running VMs on macOS. Fork & Clone the Repository ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -325,6 +287,6 @@ Qubes To configure a multi-machine evironment in Qubes, follow the Quick Start instructions above to create a standalone VM named ``sd-dev``, then follow the Linux instructions above to install the -required packages, *omitting* Virtualbox. +required packages. Then, complete the steps described in :doc:`qubes_staging`. diff --git a/docs/development/upgrade_testing.rst b/docs/development/upgrade_testing.rst index 3795b70e7..2f5b59155 100644 --- a/docs/development/upgrade_testing.rst +++ b/docs/development/upgrade_testing.rst @@ -19,10 +19,9 @@ You can use this scenario to test the upgrade process, using using either locally-built .debs or packages from the FPF test repo at https://apt-test.freedom.press/. Both options are described below. -.. note:: The upgrade scenario uses QEMU/KVM via Vagrant's libvirt provider, in - place of the default Virtualbox provider. If you haven't already done so, - you'll need to set up the libvirt provider before proceeding. For - more information, see :ref:`libvirt_provider`. +.. note:: The upgrade scenario uses QEMU/KVM via Vagrant's libvirt provider. + If you haven't already done so, you'll need to set up the libvirt provider + before proceeding. For more information, see :ref:`libvirt_provider`. .. _upgrade_testing_local: diff --git a/docs/development/virtual_environments.rst b/docs/development/virtual_environments.rst index 7363d146b..aac86f822 100644 --- a/docs/development/virtual_environments.rst +++ b/docs/development/virtual_environments.rst @@ -15,9 +15,6 @@ one. .. note:: If you plan to alter the configuration of any of these machines, make sure to review the :ref:`config_tests` documentation. -.. note:: If you see test failures due to ``Too many levels of symbolic links`` - and you are using VirtualBox, try restarting VirtualBox. - .. _staging_vms: Staging @@ -57,48 +54,6 @@ To rebuild the local packages for the app code and update the staging VMs: The Debian packages will be rebuilt from the current state of your local git repository and then installed on the staging servers. -.. note:: If you are using macOS and you run into errors from Ansible - such as ``OSError: [Errno 24] Too many open files``, you may need to - increase the maximum number of open files. Some guides online suggest - a procedure to do this that involves booting to recovery mode - and turning off System Integrity Protection (``csrutil disable``). - However this is a critical security feature and should not be - disabled. Instead follow this procedure to increase the file limit. - - Set ``/Library/LaunchDaemons/limit.maxfiles.plist`` to the following: - - .. code:: sh - - - - - - Label - limit.maxfiles - ProgramArguments - - launchctl - limit - maxfiles - 65536 - 65536 - - RunAtLoad - - ServiceIPC - - - - - The plist file should be owned by ``root:wheel``: - - .. code:: sh - - sudo chown root:wheel /Library/LaunchDaemons/limit.maxfiles.plist - - This will increase the maximum open file limits system wide on macOS - (last tested on 10.11.6). - The web interfaces and SSH are available over Tor. A copy of the the Onion URLs for *Source* and *Journalist Interfaces*, as well as SSH access, are written to the Vagrant host's ``install_files/ansible-base`` directory. @@ -173,8 +128,7 @@ playbook from running with Vagrant-specific info. You can provision production VMs from an Admin Workstation (most realistic), or from your host. If your host OS is Linux-based and you plan to use an Admin Workstation, you will need to switch Vagrant's default virtualization provider -from ``virtualbox`` to ``libvirt``. The Admin Workstation VM configuration -under Linux uses QEMU/KVM, which cannot run simultaneously with Virtualbox. +to ``libvirt``. Instructions for both installation methods follow. @@ -370,4 +324,3 @@ SSH Access By default, direct SSH access is not enabled in the prod environment. You will need to log in over Tor after initial provisioning or set ``enable_ssh_over_tor`` to "false" during ``./securedrop-admin tailsconfig``. - diff --git a/docs/development/virtualizing_tails.rst b/docs/development/virtualizing_tails.rst index 8ace62773..160489932 100644 --- a/docs/development/virtualizing_tails.rst +++ b/docs/development/virtualizing_tails.rst @@ -12,244 +12,7 @@ Tails in a virtual machine. .. _`Tails`: https://tails.boum.org -macOS ------ - -For the macOS instructions, you will use VirtualBox to create a Tails VM that -you can use to install SecureDrop on ``app-prod`` and ``mon-prod``. - -Create a VirtualBox VM -~~~~~~~~~~~~~~~~~~~~~~ - -1. Open VirtualBox -2. Click **New** to create a new VM with the following options: - - * **Name**: "Admin Workstation" - * **Type**: "Linux" - * **Version**: "Debian (64-bit)" - -.. note:: You may call the VM a different name, but you must replace - "Admin Workstation" later on in these instructions with the name you select. - -3. Click **Continue**. -4. At the prompt, configure at least 2048 MB of RAM. Click **Continue**. -5. Leave the default **Create a virtual hard disk now** selected and click - **Create**. All the default options (**Hard disk file type: VDI (VirtualBox - Disk Image)** and **Dynamically allocated**) are fine. Click **Create**. - -Booting Tails -~~~~~~~~~~~~~ - -Now that the VM is set up, you are ready to boot to Tails. Select the new VM -in the VirtualBox sidebar, and click **Settings**. - -1. Click **Storage**. -2. Click **Empty** under **Controller: IDE**. -3. Click the CD icon next to **Optical Drive:** and click **Choose Virtual - Optical Disk File**. -4. Navigate to the Tails ISO to boot from. -5. Click **General** then **Advanced**. -6. Under **Shared Clipboard** select **Bidirectional** instead of **Disabled**. - This option will enable you to transfer text from your host to the Tails VM, - which we will use later on in these steps. - - .. note:: Alternatively you can open these docs in Tor Browser in Tails. - This will obviate the need to copy and paste between the guest - and host OS. - -Install Tails -~~~~~~~~~~~~~ - -Next you will install Tails onto the Virtual Hard Disk Image. Start the VM, boot -to Tails, and enter an administration password and start Tails. - -.. note:: For all the instructions that follow, you will need to configure an - administration password each time you boot Tails. - -1. Copy the following patch and save it as ``installer.patch`` in a folder in - your Tails VM: - -.. code:: Diff - - --- /usr/lib/python2.7/dist-packages/tails_installer/creator.py 2018-01-22 14:59:40.000000000 +0100 - +++ /usr/lib/python2.7/dist-packages/tails_installer/creator.py.mod 2018-03-05 05:15:00.000000000 -0800 - @@ -595,16 +595,6 @@ class LinuxTailsInstallerCreator(TailsInstallerCreator): - self.log.debug('Skipping non-removable device: %s' - % data['device']) - - - # Only pay attention to USB and SDIO devices, unless --force'd - - iface = drive.props.connection_bus - - if iface != 'usb' and iface != 'sdio' \ - - and self.opts.force != data['device']: - - self.log.warning( - - "Skipping device '%(device)s' connected to '%(interface)s' interface" - - % {'device': data['udi'], 'interface': iface} - - ) - - continue - - - # Skip optical drives - if data['is_optical'] and self.opts.force != data['device']: - self.log.debug('Skipping optical device: %s' % data['device']) - --- /usr/lib/python2.7/dist-packages/tails_installer/gui.py 2018-01-22 14:59:40.000000000 +0100 - +++ /usr/lib/python2.7/dist-packages/tails_installer/gui.py.mod 2018-03-05 05:15:00.000000000 -0800 - @@ -568,16 +568,6 @@ class TailsInstallerWindow(Gtk.ApplicationWindow): - self.devices_with_persistence.append(info['parent']) - continue - pretty_name = self.get_device_pretty_name(info) - - # Skip devices with non-removable bit enabled - - if not info['removable']: - - message =_('The USB stick "%(pretty_name)s"' - - ' is configured as non-removable by its' - - ' manufacturer and Tails will fail to start on it.' - - ' Please try installing on a different model.') % { - - 'pretty_name': pretty_name - - } - - self.status(message) - - continue - # Skip too small devices, but inform the user - if not info['is_device_big_enough_for_installation']: - message =_('The device "%(pretty_name)s"' - -2. Now run the following two commands in a Terminal in your Tails VM: - -.. code:: sh - - sudo patch -p0 -d/ < installer.patch - sudo /usr/bin/python -tt /usr/bin/tails-installer -u -n --clone -P -m -x - -3. The **Tails Installer** will appear. Click **Install Tails**. -4. Once complete, navigate to **Applications**, **Utilities** and open **Disks**. -5. Click on the disk named "Tails" and click the Play icon to mount the disk. -6. Next open ``/media/amnesia/Tails/syslinux/live*.cfg`` and delete all instances - of ``live-media=removable``. -7. Shut down the VM. - -Boot to Tails Hard Drive Install -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Now we will remove the CD and boot to the Tails we just installed on our -virtual hard drive. From macOS you should: - -1. Click the VM in the sidebar of VirtualBox and click **Settings**. -2. Click **Storage** and select the Tails .iso under **Controller: IDE**. -3. Click the CD icon, then **Remove Disk from Virtual Drive**. -4. Click **Ok**. -5. Start the VM. - -Configure Persistence -~~~~~~~~~~~~~~~~~~~~~ - -Now in your booted Tails VM you should: - -1. Configure an admin password when prompted. -2. Copy the following patch to the Tails VM and save it as ``persistence.patch``: - -.. code:: Diff - - --- /usr/share/perl5/Tails/Persistence/Setup.pm 2017-06-30 09:56:25.000000000 +0000 - +++ /usr/share/perl5/Tails/Persistence/Setup.pm.mod 2017-07-20 07:17:48.472000000 +0000 - @@ -404,19 +404,6 @@ - - my @checks = ( - { - - method => 'drive_is_connected_via_a_supported_interface', - - message => $self->encoding->decode(gettext( - - "Tails is running from non-USB / non-SDIO device %s.")), - - needs_drive_arg => 1, - - }, - - { - - method => 'drive_is_optical', - - message => $self->encoding->decode(gettext( - - "Device %s is optical.")), - - must_be_false => 1, - - needs_drive_arg => 1, - - }, - - { - method => 'started_from_device_installed_with_tails_installer', - message => $self->encoding->decode(gettext( - "Device %s was not created using Tails Installer.")), - -3. To apply the patch, from the Terminal run: - -.. code:: sh - - sudo patch -p0 -d/ < persistence.patch - -4. Navigate to **Applications** then **Tails** and click **Configure - persistent volume**. Configure a persistent volume enabling all persistence - options. - -Shared Folders -~~~~~~~~~~~~~~ - -1. In macOS, click on the Tails VM in VirtualBox and then go to - **Settings**. -2. Click on **Shared Folders** and click the button on the right hand side to - add the folder. Navigate to the location of the SecureDrop repository on - your local machine. Check **Auto-mount**. Do not check - **Read-only**. - -3. Now reboot your Tails VM, decrypt the Persistent volume, and run the following - commands in a **Terminal** in Tails: - -.. code:: sh - - mkdir ~/Persistent/securedrop - echo 'if [ ! -d ~/Persistent/securedrop/install_files ]; then sudo mount -t vboxsf -o uid=$UID,gid=$(id -g) securedrop ~/Persistent/securedrop; fi' >> /live/persistence/TailsData_unlocked/dotfiles/.bashrc - -The first time you open a Terminal in that session you will be prompted for your -sudo password and the shared folder will be mounted. Each time you open a -Terminal thereafter in the Tails session, your sudo password will not be needed. - -Allow the Guest to Create Symlinks -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Finally, you'll need to allow the guest to create symlinks, which are -`disabled by default in VirtualBox`_. - -.. _`disabled by default in VirtualBox`: https://www.virtualbox.org/ticket/10085#comment:12 - -Shut down the Tails VM, and in your host run: - -.. code:: sh - - VBoxManage setextradata "Admin Workstation" VBoxInternal2/SharedFoldersEnableSymlinksCreate/securedrop 1 - -.. note:: If you named your Tails VM something other than "Admin Workstation", - you can run ``VBoxManage list vms`` to get the name of the Virtual Machine. - -Finally, restart VirtualBox. - -Configure Networking -~~~~~~~~~~~~~~~~~~~~ - -In order to communicate with the server VMs, you'll need to attach this -virtualized *Admin Workstation* to the ``securedrop`` network. - -.. warning:: If you named the SecureDrop repository something other than - ``securedrop``, you should connect your VM to the network of the same name. - -With the *Admin Workstation* VM turned off, you should: - -1. Click on the VM in VirtualBox. -2. Click **Settings**. -3. Click **Network** and then **Adapter 2**. -4. Enable this network adapter and attach it to the **Internal Network** called - ``securedrop``. -5. Click OK and start the VM. - -Now you should be able to boot to Tails, decrypt the Persistent volume, -navigate to ``~/Persistent/securedrop`` and proceed with the :ref:`production -install `. - -Disable Shared Clipboard (Optional) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1. Click on the VM in VirtualBox. -2. Click **Settings**. -3. Click **General** and then **Advanced**. -4. Now that you are finished with copy pasting the patches above you can change - the **Shared Clipboard** from **Bidirectional** back to **Disabled**. +Only libvirt-based virtualization, on a Linux host, is supported. Linux -----