Create VM provisioning utility script

[conorsch]

Errors in the Salt provisioning flow (e.g. make all) often do not percolate up to cause the make invocation to exit non-zero. This means that it can look like the testing VMs were created successfully, but some actually lack required files to operate as part of the SD workstation environment.

One such example occurred during review of #47, when configuring the sd-decrypt:

Summary for local
-----------
Succeeded: 26 (changed=5)
Failed: 0
-----------
Total states run: 26
Total run time: 5.960 s
sd-decrypt: ERROR (exception [('/srv/salt/_tops/base/sd-dispvm.top', '/tmp/tmpzqv1xs/srv/salt/_tops/sd-dispvm.top', "[Errno 2] No such file or directory: '/srv/salt/_tops/base/sd-dispvm.top'"), ('/srv/salt/_tops/base/sd-dispvm-files.top', '/tmp/tmpzqv1xs/srv/salt/_tops/base/sd-dispvm-files.top', "[Errno 2] No such file or directory: '/srv/salt/_tops/base/sd-dispvm-files.top'")])
sudo qubesctl top.enable sd-svs-disp

Note that I transcribed the error output above by hand, due to lack of copy/paste support from dom0, so there may be typos. The problem isn't so much the particular error, more that the sd-decrypt VM was not fully configured, and yet make all exited zero, implying no problems.

[redshiftzero]

Indeed, currently make all returns 1 only in a subset of possible failure scenarios, e,g, if /srv/salt/sd is already populated it will return 1. I hit the same issue as you in #72: VMs were created, but some critical files were not created, but make all returned 0.

While the Makefile is definitely super handy during development, I think what we'd eventually want for provisioning in dom0 is a Python script that can do e.g. the following:

securedrop-vm-manager create all
securedrop-vm-manager delete all
securedrop-vm-manager create <vm-name>
securedrop-vm-manager delete <vm-name>

The target user here would be the admin but of course this would be useful for development as well (we'd want a --verbose, -v flag that would spit out all output for debugging issues).

This would make it easier for us to exert more control over the provisioning process, ensuring that each step executes, and produces the appropriate return code. For robustness, after provisioning each VM we could also explicitly check (instead of relying only on salt) that necessary files were created and return 1 with a descriptive message if a file was not created. For example, for my issue in #72, the script could do (this is in dom0):

qvm-run sd-whonix 'tail /etc/tor/torrc' --pass-io

and parse the output - if the string HidServAuth is not found, we could return 1 and display to the user ERROR during creation of sd-whonix: HidServAuth not found in /etc/tor/torrc. Since this is the purpose of what some of the unit and integration tests are currently doing, we could potentially reuse parts of that code in such a provisioning script.

Thoughts welcome

[joshuathayer]

I agree, having a script to manage SD VMs (including a more robust build process) is a great idea. It could be extended to include debugging and administrative functions like checking the version of the installed SD software, check the signature of the key in the GPG VM, etc.

This may also inform some thinking about #73.

[conorsch]

We certainly have a "VM provisioning utility script" these days, namely sdw-admin, which is used in all environments—dev, staging, and prod—to handle creation and configuration of VMs as part of installation, or state enforcement. There's also the "handle-upgrade" script which has been helpful in managing transitions between Fedora template versions. I don't see any outstanding need for further tooling, but if someone does, please reopen.