Support in-place upgrades for Focal VMs

[emkll]

Description

We currently have a (libvirt-only) upgrade molecule scenario that is updated at every SecureDrop release that can be use to test upcoming SecureDrop versions automatically.

We should add a new molecule scenario for upgrade-focal, upload new focal boxes new boxes on the latest release that supports Focal, and ensure the upgrade is working

[eloquence]

This will be needed for upgrade-testing between Focal releases, so can wait a bit longer.

[eloquence]

Discussed with @emkll in backlog review, we would suggest to do this only for Focal and, at the same time, remove the Xenial logic for it, since it is not critically required for QA. This may also be an opportunity to add this test as a nightly or weekly CI check, for example. Thoughts from others?

[conorsch]

Took a stab at this today. Pushed a WIP branch to https://github.com/freedomofpress/securedrop/tree/5512-focal-upgrade-scenario, which includes updates to the build logic for preparing the Focal upgrade VMs, but when running make upgrade-start with them pulled in, the VMs fail to boot with a grub error:

With more work, it's likely we'll be able to suss out that disk image problem and get it working. For 1.8.1, though, we're getting close to QA already. Given the breakage documented in #5883, for the purposes of 1.8.1 upgrade QA, I suggest we use the vagrant-based logic introduced in https://github.com/freedomofpress/securedrop/pull/5669/files#diff-f54b821e3e899deafaf7c25d6c3f694a9a8334abc57bcc6f0ea0a9282e7790f9R6-R13. With a bit of documentation, that should address the immediate need to perform in-place updates (via unattended-upgrades) on Focal VMs, with locally built packages.

[conorsch]

As mentioned above:

I suggest we use the vagrant-based logic introduced in https://github.com/freedomofpress/securedrop/pull/5669/files#diff-f54b821e3e899deafaf7c25d6c3f694a9a8334abc57bcc6f0ea0a9282e7790f9R6-R13. With a bit of documentation, that should address the immediate need to perform in-place updates (via unattended-upgrades) on Focal VMs, with locally built packages.

That seems like the best approach forward here. Specifically, that means:

• removing the upgrade molecule scenario
• removing the vagrant_packager molecule scenario
• no longer maintaining the "upgrade" boxes in s3
• when we want to test upgrades, we'd use a solution like apt-local, which ideally would replace the current molecule scenario

If folks have objections to the above, please speak up, otherwise I'll proceed with removing those bits and updating the docs to focus on a leaner procedure. Hopefully soon we can have a larger conversation about how we test, and what specific procedures we mandate for QA (whether hardware or VM based), and we can design the upgrade testing procedures around those expectations.

[conorsch]

Only just started on implementation work today, so didn't get very far. The list of goals above remains solid, although I'll add that the current "apt-local" setup is split between having requirements in the Vagrantfile and in the molecule/upgrade/ directory. We should disentangle those a bit, and make it more straightforward to create prod libvirt VMs, specifically for use with the Tails environment, with or without upgrade testing capability. That may involve resolving #3208, but it not be worth the time for these changes. Started a WIP branch here: https://github.com/freedomofpress/securedrop/tree/5512-remove-upgrade-scenario

Once the new consolidated workflow has been tested, based on changes in that branch, I'll work on a docs-only PR documenting same, and submit together for review.