[Qubes 4.2] Test an SDW installation against development build

[rocodes]

Tracking issue to document testing clean install of SDW against Qubes 4.2 build

Known to-dos:

• Build fedora-37 version of dom0 packages, including the base template
• Split templates into their own repo

Potential to-dos:

• Migrate to Debian Bookworm

[eloquence]

I've begun installation of a weekly from https://qubes.notset.fr/iso-testing/ (per this thread) on a T480. Make sure you have the qubes-weekly-builds-signing-key.asc from the same directory (fingerprint should be 9B7E61D3BB70C4B1335CE5B67B72A119CCCA57BB) installed if you want to verify the signature.

Will poke a bit more tomorrow.

[eloquence]

First observations:

• 4.2 uses fedora-37 as the base template in dom0, so as may be expected we'll need to build for a new release. Our fedora-32 RPM is installable in dom0, however.
• Seeing a lot of very disruptive setuptools is replacing distutils deprecation warnings all over the place, not sure yet if that's due to anything on our end or if it's just the Python env in dom0
• Install aborts early because fedora-37 template is not yet available in https://yum.qubes-os.org/r4.2/templates-itl/rpm/, will try installing it from testing or patch to downgrade to fedora-36

[eloquence]

Enabling the testing repo in /etc/qubes/repo-templates/qubes-templates.repo resolved the issue of installing fedora-37; however, our f32 SDW template does not appear to be installable via qvm-template after https://yum.securedrop.org/workstation/dom0/f32/ is configured as a repo.

The template installation stalls and fails with Error canonicalizing file: failed to fill whole buffer. This occurs during the use of rpmcanon, a Rust tool Qubes uses to mitigate security risks from installing RPMs (https://github.com/QubesOS/qubes-rpm-oxide/blob/main/rpm-writer/bin/rpmcanon.rs and QubesOS/qubes-issues#6485).

This could also simply be because our existing published template is only built for f32 right now, but CC-ing @marmarek in case there other potential reasons for this error that are worth investigating.

[marmarek]

In theory, it should work I think. @DemiMarie, any idea?

[eloquence]

Unrelated to above, I explored the new GUI updater in a bit more detail today and documented some screenshots and initial observations in this (publicly viewable) GDoc.

[DemiMarie]

@marmarek yup, it should work.

[DemiMarie]

@eloquence why are you still using Fedora 32? It’s bee end of life for a long long time.

[marmarek]

@eloquence why are you still using Fedora 32? It’s bee end of life for a long long time.

I believe he meant dom0 packages built for R4.1.

[DemiMarie]

@eloquence why are you still using Fedora 32? It’s bee end of life for a long long time.

I believe he meant dom0 packages built for R4.1.

Ah, fair.

[eaon]

First note on the Fedora 37 template situation:

The Fedora 37 template isn't available in the 4.2 stable template repos but current 4.2 builds ship a Fedora 37 template. Unless Fedora 38 gets shipped with 4.2 when it's being released, it wouldn't even be downloaded from there. Why does this matter? Our Salt logic doesn't check if a template is installed separate from the qvm-template install $template attempt, but it assumes that if that process comes back with return code 0 that it's installed. Because qvm-template itself doesn't check whether a template is installed before network requests are being made, that salt state fails on when the template is installed but technically isn't available to download.

IMO, either we, or qvm-template should check the existing templates before looking upstream. This would also help us with testing our own templates during development.

[eloquence]

(FYI, I've started building out a checklist in the main body of this issue, pls feel free to edit)

[eloquence]

Michael pointed out that it's possible to install our template from disk (just simply via qvm-template install <filename>), and that works with the old template. I think that's because it bypasses the rpmcanon check which is performed when downloading via the network:
https://github.com/QubesOS/qubes-core-admin-client/blob/c70da44702109b45d960c7de170ab8c2820b7167/qubesadmin/tools/qvm_template.py#L593

After that, sdw-admin --apply fails for me due to certain RPC policy files no longer being present in 4.2:

• qubes.Gpg
• qubes.GpgImportKey
• qubes.ClipboardPaste

[marmarek]

After that, sdw-admin --apply fails for me due to certain RPC policy files no longer being present in 4.2:

Indeed most policies were transferred to the new location at /etc/qubes/policy.d. I strongly recommend you use the new location too (a single file for all relevant services or such). It works on R4.1 too.

[eloquence]

Poked a bit more at the updater experience today.

• Looks like udpate timestamps get written to /var/lib/qubes/qubes.xml and the update GUI mostly calls out to the new qubes-vm-update tool to do its thing.
• I've not seen update timestamps for dom0 ever successfully get written, filed [4.2-weekly] Updater never records dom0 update time QubesOS/qubes-issues#8240 for that
• I'm not quite sure how the staleness check is intended to work yet. From what I can tell, it is supposed to pre-select VMs for updating if they've not been updated in a while (to be determined by a user preference). However, for VMs that have never been updated, I only see it tagging them with MAYBE and not pre-selecting them in the GUI.

[eaon]

I can confirm that dom0's last updated/checked isn't being set.

While /var/lib/qubes/qubes.xml is a file that is being written to, as far as I can tell the better system to ask about the update status is the Features subsystem: qvm-features securedrop-workstation-bullseye last-updates-check works, however last-update isn't available (even though I did update the template) - it works with whonix-ws-16 though. Not entirely sure why yet.

If I read qui/updater/updater_settings.py correctly, the staleness check can be configured per-template (or vm) - which is perfect for us. Where I'm not entirely clear yet is dom0 seems to stand in for the default setting, which isn't quite how we've treated this situation in our own updater. If we continue to enforce dom0 updates, we would inadvertently enforce update checks for all templates.

[eaon]

Seeing a lot of very disruptive setuptools is replacing distutils deprecation warnings all over the place, not sure yet if that's due to anything on our end or if it's just the Python env in dom0

I believe that's due to this: QubesOS/qubes-issues#8079 (comment)

@marmarek, do I understand that right that in Python 3.11 land (i.e. dom0) we may be stuck with the warning?

[eloquence]

Unless Fedora 38 gets shipped with 4.2 when it's being released,

That's the case now; 4.2RC1 ships with fedora-38. I pushed a small commit to revise but if it gets in your way feel free to drop it!

[eloquence]

No change in RC1 when attempting to install our old f32 template via download (after temporarily updating the dom0 template repo config to point to f32), behavior is the same as in weeklies, including on repeated runs:

[root@dom0 repo-templates]# qvm-template install securedrop-workstation-bullseye
Downloading 'qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135'...
qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135:   0%|                                                                                         | 0.00/895M [00:00<?, ?B/s]Error canonicalizing file: failed to fill whole buffer
qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135:   0%|                                                                                         | 0.00/895M [00:00<?, ?B/s]
'qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135' download failed, retrying...
qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135:   0%|                                                                                         | 0.00/895M [00:00<?, ?B/s]Error canonicalizing file: failed to fill whole buffer
qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135:   0%|                                                                                         | 0.00/895M [00:00<?, ?B/s]
'qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135' download failed, retrying...
qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135:   0%|                                                                                         | 0.00/895M [00:00<?, ?B/s]Error canonicalizing file: failed to fill whole buffer
qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135:   0%|                                                                                         | 0.00/895M [00:00<?, ?B/s]
'qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135' download failed, retrying...
qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135:   0%|                                                                                         | 0.00/895M [00:00<?, ?B/s]Error canonicalizing file: failed to fill whole buffer
qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135:   0%|                                                                                         | 0.00/895M [00:00<?, ?B/s]
'qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135' download failed, retrying...
qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135:   0%|                                                                                         | 0.00/895M [00:00<?, ?B/s]Error canonicalizing file: failed to fill whole buffer
qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135:   0%|                                                                                         | 0.00/895M [00:00<?, ?B/s]
Error: 'qubes-template-securedrop-workstation-bullseye-0:4.0.6-202206302135' download failed.

Will proceed to again download manually & then install.

[DemiMarie]

@eloquence: nobody should be using a Fedora 32 template.

[DemiMarie]

All packages for R4.2 dom0 will need to be recompiled.

[eloquence]

@eloquence: nobody should be using a Fedora 32 template.

@DemiMarie Yup, that's understood. Until we've got a new template ready for testing, we can make it work by manually downloading and installing the f32 one (only during development).

[eloquence]

With @eaon's changes in skip-ci/884-qubes-4.2 branch, I was able to get the SecureDrop Worsktation provisioned, and to log into the SecureDrop Client (after manually installing the downloaded f32 template as noted above). Syncing with the server and opening files in disposable VMs works:

Reply/export currently trigger RPC policy denial errors and fail for that reason. However, I was able to export files successfully by manually attaching a device to sd-devices.

[eloquence]

I've added some further notes of my understanding re: current SDW updater behavior, 4.2 updater behavior, and decision tree questions for consolidation to the working doc. Will go over these with Michael for fact-checking/expansion before fleshing out in issue form.

[zenmonkeykstop]

Cory and I will review said doc, once other sprint deliverables are in the bag, and manage conversation re: next steps.

[eloquence]

Just noting here as well that 4.2 will likely ship with Whonix 17, we may see some additional breakage there once it lands in an RC.

[eloquence]

Quick notes based on latest RC:

• 4.2 is indeed on Whonix 17 now.
• Default templates have been renamed, which will necessitate provisioning code updates:
  • debian-12 -> debian-12-xfce
  • fedora-38 -> fedora-38-xfce
  • whonix-gw-* -> whonix-gateway-*
  • whonix-ws-* -> whonix-workstation-*
• Updater is working more reliably for me now including for Whonix. I filed two issues: [4.2] Even after update, Debian template shows up as "unknown" for last updated status QubesOS/qubes-issues#8687 and [4.2] When "restart" option is set in updater, VMs should be restarted without addtional user selection QubesOS/qubes-issues#8686

[zenmonkeykstop]

Updated fedora-38 and whonix-* naming as per above, made a little progress on getting stuff working on 4.2 again. Currently blocked by the lack of a f37 version of the securedrop-workstation-bullseye template, which probably means we now have to either:

• create a (bookworm, ideally) version of this template and add it in f37/ on yum-test, or
• proceed with the plan to drop it in favour of deriving straight from debian-12 as per Consider preferring Salt customizations to custom template RPM #765 (comment)

[zenmonkeykstop]

(Can temporarily work around the above by setting the default fedora_repo to f32 instead of f37 in dom0/sd-default-config.sls.)

[deeplow]

Default templates have been renamed, which will necessitate provisioning code updates:

debian-12 -> debian-12-xfce
fedora-38 -> fedora-38-xfce

For context, this is not just a rename. The debian-12 and fedora-38 templates still do exist. They are just not installed by default as they're gnome-based. Instead, the XFCE templates have now become the default as a result of this poll.

IMO this has usability implications.

[zenmonkeykstop]

#970 is under review, we can close this.