Update Fedora TemplateVM on package install #558

emkll · 2020-05-21T14:46:02Z

Status

Ready for review

Description of Changes

Towards #544

Updates workstation-default Fedora TemplateVM if and only if the TemplateVM package has just been installed in dom0

Testing

Should we increment to 0.3.0-rc2?

Clean Install

This requires the sys vms to be using Fedora-30 TemplateVMs, and for the Fedora-31 TemplateVM package to not be installed in dom0

make all on this branch completes successfully
make test all tests pass
sudo dnf update in fedora-31 indicates no updates are required
Running the updater does update TemplateVM as part of the first step(this will always happen, regardless of update status due to Optimize Fedora Template updates #486), but the dom0 step does not trigger the update
sys-* VMS use fedora-31 as their template

Upgrade testing (I have tested this is a prod environment)

Install 0.2.4 RPM
Install RPM from this branch
Run GUI updater (time /opt/securedrop/launcher/sdw-launcher.py --skip-delta 0)
GUI completes successfully after 20-25 minutes
Fedora-31 is installed
sudo dnf update in fedora-31 indicates no updates are required
Running the updater does update TemplateVM as part of the first step(this will always happen, regardless of update status due to Optimize Fedora Template updates #486), but the dom0 step does not trigger the update
sys-* VMS use fedora-31 as their template

Checklist

If you have made code changes

Linter (make flake8) passes in the development environment (this box may
be left unchecked, as flake8 also runs in CI)

If you have made changes to the provisioning logic

All tests (make test) pass in dom0 of a Qubes install

…nager In the Preflight updater, bacause the TemplateVM updates occur before dom0 state is applied, the Fedora-31 TemplateVM is not updated as it is installed. It will require another update cycle (8+h in order to get updated again). Here, we should simply ensure the default Fedora TemplateVM is updated, if and only if the TemplateVM was just installed.

zenmonkeykstop · 2020-05-21T16:30:28Z

Testing upgrade against a pre-existing 0.24 install.

zenmonkeykstop · 2020-05-21T21:15:58Z

built rpm and transferred to a 0.24 prod workstation, applied with sudo dnf install <nameofrpm>
ran updater, accepted updates
updater ran for ~2h30, failed with errors unrelated to fedora-31, rebooted, then ran again with same errors:

2020-05-21 16:41:07,233 - sdw_updater_gui.UpdaterApp:194(apply_all_updates) INFO: Starting UpgradeThread
2020-05-21 16:41:07,242 - sdw_updater_gui.Updater:80(apply_updates) INFO: Applying all updates
2020-05-21 16:41:07,243 - sdw_updater_gui.Updater:181(_apply_updates_vm) INFO: Updating fedora:fedora-31
2020-05-21 16:42:31,849 - sdw_updater_gui.Updater:202(_apply_updates_vm) INFO: fedora-31 update successful
2020-05-21 16:42:31,849 - sdw_updater_gui.Updater:181(_apply_updates_vm) INFO: Updating sd-gpg:securedrop-workstation-buster
2020-05-21 16:42:31,855 - sdw_updater_gui.UpdaterApp:173(update_progress_bar) INFO: Signal: Progress 45%
2020-05-21 16:43:55,708 - sdw_updater_gui.Updater:197(_apply_updates_vm) ERROR: An error has occurred updating securedrop-workstation-buster. Please contact your administrator.
2020-05-21 16:43:55,708 - sdw_updater_gui.Updater:200(_apply_updates_vm) ERROR: Command '['sudo', 'qubesctl', '--skip-dom0', '--targets', 'securedrop-workstation-buster', 'state.sls', 'update.qubes-vm']' returned non-zero exit status 20
2020-05-21 16:43:55,708 - sdw_updater_gui.Updater:386(apply_dom0_state) INFO: Applying dom0 state
2020-05-21 16:43:55,709 - sdw_updater_gui.UpdaterApp:173(update_progress_bar) INFO: Signal: Progress 95%
2020-05-21 16:44:20,610 - sdw_updater_gui.Updater:389(apply_dom0_state) INFO: Dom0 state applied
2020-05-21 16:44:20,610 - sdw_updater_gui.Updater:421(shutdown_and_start_vms) INFO: Shutting down SDW TemplateVMs for updates
2020-05-21 16:44:21,796 - sdw_updater_gui.Updater:425(shutdown_and_start_vms) INFO: Shutting down SDW AppVMs for updates
2020-05-21 16:44:49,735 - sdw_updater_gui.Updater:433(shutdown_and_start_vms) INFO: Safely shutting down system VM: sys-usb
2020-05-21 16:44:55,984 - sdw_updater_gui.Updater:433(shutdown_and_start_vms) INFO: Safely shutting down system VM: sys-whonix
2020-05-21 16:45:00,569 - sdw_updater_gui.Updater:440(shutdown_and_start_vms) INFO: Killing system VM: sys-firewall
2020-05-21 16:45:04,042 - sdw_updater_gui.Updater:440(shutdown_and_start_vms) INFO: Killing system VM: sys-net
2020-05-21 16:45:09,054 - sdw_updater_gui.Updater:449(shutdown_and_start_vms) INFO: Starting fedora-based system VMs after updates
2020-05-21 16:45:09,210 - sdw_updater_gui.Updater:473(_safely_start_vm) INFO: VMs running before start of sys-net: b'dom0\n'
2020-05-21 16:45:22,191 - sdw_updater_gui.Updater:473(_safely_start_vm) INFO: VMs running before start of sys-firewall: b'dom0\nsys-net\n'
2020-05-21 16:45:36,674 - sdw_updater_gui.Updater:473(_safely_start_vm) INFO: VMs running before start of sys-whonix: b'dom0\nsys-firewall\nsys-net\n'
2020-05-21 16:45:45,276 - sdw_updater_gui.Updater:473(_safely_start_vm) INFO: VMs running before start of sys-usb: b'dom0\nsys-firewall\nsys-net\nsys-whonix\n'
2020-05-21 16:46:04,164 - sdw_updater_gui.Updater:453(shutdown_and_start_vms) INFO: Starting SDW VMs after updates
2020-05-21 16:46:04,581 - sdw_updater_gui.Updater:473(_safely_start_vm) INFO: VMs running before start of sd-log: b'dom0\nsys-firewall\nsys-net\nsys-usb\nsys-whonix\n'
2020-05-21 16:46:27,677 - sdw_updater_gui.Updater:473(_safely_start_vm) INFO: VMs running before start of sd-gpg: b'dom0\nsd-log\nsys-firewall\nsys-net\nsys-usb\nsys-whonix\n'
2020-05-21 16:46:49,859 - sdw_updater_gui.Updater:473(_safely_start_vm) INFO: VMs running before start of sd-whonix: b'dom0\nsd-gpg\nsd-log\nsys-firewall\nsys-net\nsys-usb\nsys-whonix\n'
2020-05-21 16:47:02,695 - sdw_updater_gui.Updater:473(_safely_start_vm) INFO: VMs running before start of sd-proxy: b'dom0\nsd-gpg\nsd-log\nsd-whonix\nsys-firewall\nsys-net\nsys-usb\nsys-whonix\n'
2020-05-21 16:47:30,671 - sdw_updater_gui.Updater:473(_safely_start_vm) INFO: VMs running before start of sd-app: b'dom0\nsd-gpg\nsd-log\nsd-proxy\nsd-whonix\nsys-firewall\nsys-net\nsys-usb\nsys-whonix\n'
2020-05-21 16:47:54,351 - sdw_updater_gui.Updater:248(_write_updates_status_flag_to_disk) INFO: Setting update flag to 3 in sd-app
2020-05-21 16:47:58,411 - sdw_updater_gui.Updater:261(_write_updates_status_flag_to_disk) INFO: Setting update flag to 3 in dom0
2020-05-21 16:47:58,413 - sdw_updater_gui.UpdaterApp:130(upgrade_status) INFO: Signal: upgrade_status {'recommended_action': <UpdateStatus.UPDATES_FAILED: '3'>, 'fedora': <UpdateStatus.UPDATES_OK: '0'>, 'apply_dom0': '0', 'sd-gpg': <UpdateStatus.UPDATES_FAILED: '3'>}
2020-05-21 16:47:58,416 - sdw_updater_gui.UpdaterApp:153(upgrade_status) INFO: Error upgrading VMs

fedora-31 is installed
sudo dnf update indicates nothing to do!
fixed securedrop-workstation-buster by running sudo dkpg --configure -a against it
reran preflight updater, check phase first
- fedora status is UPDATES_REQUIRED
started updater update phase
- fedora31 update triggered but nothing changed
- dom0 step does not trigger templateVM update

emkll · 2020-05-22T13:37:00Z

Thanks @zenmonkeykstop the error you observed does indeed seem unrelated, did you observe any Failed to start securedrop-workstation-buster type messages? And all your sys VMs are running Fedora-31 after the updater, right?

zenmonkeykstop · 2020-05-22T15:55:22Z

I don't recall any notifications failures to start securedrop-workstation-buster, but I could have missed them. Some failures loading kernel modules according to console logs:

● systemd-modules-load.service - Load Kernel Modules
   Loaded: loaded (/lib/systemd/system/systemd-modules-load.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2020-05-22 11:45:31 EDT; 16s ago
     Docs: man:systemd-modules-load.service(8)
           man:modules-load.d(5)
  Process: 188 ExecStart=/lib/systemd/systemd-modules-load (code=exited, status=1/FAILURE)
 Main PID: 188 (code=exited, status=1/FAILURE)

May 22 11:45:31 localhost systemd-modules-load[188]: Inserted module 'xen_evtchn'
May 22 11:45:31 localhost systemd-modules-load[188]: Failed to find module 'xen-blkback'
May 22 11:45:31 localhost systemd-modules-load[188]: Inserted module 'xen_gntalloc'
May 22 11:45:31 localhost systemd-modules-load[188]: Inserted module 'xen_gntdev'
May 22 11:45:31 localhost systemd-modules-load[188]: Inserted module 'u2mfn'
May 22 11:45:31 localhost systemd-modules-load[188]: Failed to find module 'xen-blkback'
May 22 11:45:31 localhost systemd-modules-load[188]: Failed to find module 'xen-netback'
May 22 11:45:31 localhost systemd[1]: systemd-modules-load.service: Main process exited, code=exited, status=1/FAILURE
May 22 11:45:31 localhost systemd[1]: systemd-modules-load.service: Failed with result 'exit-code'.
May 22 11:45:31 localhost systemd[1]: Failed to start Load Kernel Modules.

Non-whonix sys-* VMs are all fedora-31 now, fedora-31 is default template (default dispvm template is sd-viewer).

conorsch · 2020-05-22T19:41:28Z

Ran through an upgrade of staging, from 0.2.4. Unfortunately I also encountered a failure on a non-Fedora VM, the libxenlight error (#498) with "request refused" in syslog (#519 (comment)).

While I'm not convinced the failure was caused by the logic introduced here, I do think more testing is required prior to merge. Even in spite of the failure, the sys-* VMs were updated to F31, which is great, but of course the updater declared failure.

On the subject of "if-and-only-if" F31 is installed does the package-upgrade logic run, that certainly appears to be the case, although the if-and-only-if behavior of cmd.wait-with-watch is in direct contradiction to the Salt docs. See table in requisites overview, as well as this note from the watch docs

If a state should only execute when another state has changes, and otherwise do nothing, the new onchanges requisite should be used instead of watch. watch is designed to add additional behavior when there are changes, but otherwise the state executes normally.

I haven't actually tested multiple runs of the full updater with the new logic to confirm that it only runs once, but brief checks with test=True indicated the package-upgrade logic wouldn't be rerun, which is the behavior we want, although still a bit confusing, given the divergence from documented behavior. Either way, we likely want to shift to cmd.run -> onchanges, to make the code as future-proof as possible.

emkll · 2020-05-22T21:16:37Z

@conorsch I had originally pursed the onchanges route (see below for diff), following the docs [1] but that did not work, the action to upgrade the template is never fired using that method.

[1] https://docs.saltstack.com/en/master/ref/states/requisites.html#state-target-matching

diff --git a/dom0/sd-sys-vms.sls b/dom0/sd-sys-vms.sls
index 87ae029..4aa63cf 100644
--- a/dom0/sd-sys-vms.sls
+++ b/dom0/sd-sys-vms.sls
@@ -19,12 +19,11 @@ dom0-install-fedora-template:
 
 # If the VM has just been installed via package manager, update it immediately
 update-fedora-template-if-new:
-  cmd.wait:
-    - name: sudo qubesctl --skip-dom0 --targets {{ sd_supported_fedora_version }} state.sls update.qubes-vm
-    - require:
-      - pkg: dom0-install-fedora-template
-    - watch:
+  cmd.run:
+    - name: sudo qubesctl --show-output state.highstate
+    - onchanges:
       - pkg: dom0-install-fedora-template
+
 # qvm.default-dispvm is not strictly required here, but we want it to be
 # updated as soon as possible to ensure make clean completes successfully, as
 # is sets the default_dispvm to fedora-31-dvm

conorsch · 2020-05-22T23:12:20Z

Ran through the test plan again, with positive results. Took a while, because I had to revert my staging instance back to baseline.

When reporting the results of ad-hoc tests against SecureDrop Workstation, please include the following information:

Environment:

Environment type: Qubes staging
Client version: [did not perform interactive client testing]
Workstation version: Built from this PR, i.e. bb74dac8638dda91ad3492af8f0bf57686b6d491
Server version: [did not perform interactive client testing]

Test steps:

First, reverted customizations from previous round of testing:

Set all sys-* VM templates to fedora-30 [sic]
qubes-prefs default_template fedora-30
sudo dnf remove qubes-template-fedora-30
sudo dnf install ./<local version of 0.2.3 RPM
sudo qubes-dom0-update --action="clean all"
sudo qubes-dom0-update
Reboot

Then, I followed the test plan provided in the OP.

Test results:

Install 0.2.4 RPM
Install RPM from this branch
Run GUI updater (time /opt/securedrop/launcher/sdw-launcher.py --skip-delta 0)
GUI completes successfully after 20-25 minutes
- Interestingly enough, clocked in at 24m44s!
Fedora-31 is installed
sudo dnf update in fedora-31 indicates no updates are required
🟠 Running the updater does update TemplateVM as part of the first step(this will always happen, regardless of update status due to Optimize Fedora Template updates #486), but the dom0 step does not trigger the update
- Didn't sufficiently validate this, now that I've got the custom RPM installed, happy to take another look.
- Running sudo qubesctl --show-output state.sls sd-sys-vms directly did not rerun the update logic, which is a decent, but not perfect, test
sys-* VMS use fedora-31 as their template

zenmonkeykstop · 2020-05-23T00:34:21Z

Ran through a fresh prod install using the RPM from this branch:

Environment:

Environment type: Qubes prod
Client version: [not tested]
Workstation version: Built from this PR
Server version: [ not tested ]

Test steps:

built sdw RPM from this branch
installed Qubes 4.0.3,
went through SDW prod install, substituting RPM built previously for the current release version

Test results:

Fedora-31 is installed
sudo dnf update in fedora-31 indicates no updates are required
sys-* VMs use fedora-31 as their template
ran preflight updater via desktop icon:
- Running the updater does update TemplateVM as part of the first step, but the dom0 step does not trigger the update

conorsch

Given two positive reviews, approving for merge. We'll proceed with manual QA once rc2 is built and uploaded.

…e-on-package-install Update Fedora TemplateVM on package install

emkll requested review from conorsch and rmol May 21, 2020 14:46

SecureDrop Workstation 0.3.0-rc2

bb74dac

conorsch approved these changes May 26, 2020

View reviewed changes

conorsch merged commit 66e709f into master May 26, 2020

emkll mentioned this pull request May 27, 2020

Update Fedora-30 based VMs to Fedora 31 #544

Closed

emkll deleted the 544-update-fedora-template-on-package-install branch July 23, 2020 14:57

cfm pushed a commit that referenced this pull request Apr 1, 2024

Merge pull request #558 from freedomofpress/544-update-fedora-templat…

71dbe4a

…e-on-package-install Update Fedora TemplateVM on package install

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Fedora TemplateVM on package install #558

Update Fedora TemplateVM on package install #558

emkll commented May 21, 2020 •

edited by zenmonkeykstop

Loading

zenmonkeykstop commented May 21, 2020

zenmonkeykstop commented May 21, 2020 •

edited

Loading

emkll commented May 22, 2020

zenmonkeykstop commented May 22, 2020

conorsch commented May 22, 2020

emkll commented May 22, 2020

conorsch commented May 22, 2020

zenmonkeykstop commented May 23, 2020 •

edited

Loading

conorsch left a comment

Update Fedora TemplateVM on package install #558

Update Fedora TemplateVM on package install #558

Conversation

emkll commented May 21, 2020 • edited by zenmonkeykstop Loading

Status

Description of Changes

Testing

Clean Install

Upgrade testing (I have tested this is a prod environment)

Checklist

If you have made code changes

If you have made changes to the provisioning logic

zenmonkeykstop commented May 21, 2020

zenmonkeykstop commented May 21, 2020 • edited Loading

emkll commented May 22, 2020

zenmonkeykstop commented May 22, 2020

conorsch commented May 22, 2020

emkll commented May 22, 2020

conorsch commented May 22, 2020

Environment:

Test steps:

Test results:

zenmonkeykstop commented May 23, 2020 • edited Loading

Environment:

Test steps:

Test results:

conorsch left a comment

Choose a reason for hiding this comment

emkll commented May 21, 2020 •

edited by zenmonkeykstop

Loading

zenmonkeykstop commented May 21, 2020 •

edited

Loading

zenmonkeykstop commented May 23, 2020 •

edited

Loading