Support for Qubes 4.1

[eloquence]

Qubes 4.1 is not out and no release date has been announced yet. However, given the scope of expected changes, we should begin testing early to ensure a smooth update for SecureDrop Workstation users. Test builds can be downloaded from OpenQA, e.g., https://openqa.qubes-os.org/assets/iso/Qubes-4.1-20200726-x86_64.iso

Major changes that have been announced include:

• changes to the Qrexec policy system, including a new policy format, performance improvements, support for socket services, and policy notifications;
• an experimental and optional GUI domain ("a qube separate from dom0 that handles all the display-related tasks and some system management")
• upgrade from fedora-25 to fedora-32 in dom0

[eloquence]

During the 8/6-8/20 sprint, @emkll has offered to take a test build for a spin, and @rmol has previously suggested doing the same. We've allocated some time to the sprint to capture initial findings, so we can get a better sense of the scope of changes that will be needed for the SecureDrop Workstation codebase.

[eloquence]

We've not made progress on this yet, but both @rmol and @emkll want to poke at this during the 8/20-9/2 sprint. Beyond installing Qubes 4.1, this may include attempting to build/install the RPM if time permits.

[conorsch]

Although we've discussed it several times, it's worth reiterating that running an early build of Qubes 4.1 will prohibit development and testing of the Workstation on that hardware, since the repository channels and several APIs have changed significantly. That's OK, though, and precisely the info we hope to gather by running it.

Rather than installing fresh from the ISO, it might be worthwhile to test-drive the burgeoning plans for an in-place upgrade: QubesOS/qubes-issues#5685 , so we can share test results with upstream. As always, backups strongly advised. 😄

[emkll]

Timeboxed make dev in a freshly installed Qubes 4.1 install (using the ISO in https://openqa.qubes-os.org/tests/10827/asset/iso/Qubes-4.1-20200726-x86_64.iso)

Here were the steps needed in order to unblock the install:

• make was not present in dom0: sudo qubes-dom0-update make
• Set sd_supported_fedora_version to fedora-32 in dom0/sd-sys-vms.sls
• removed some blocks in dom0/sd-dom0-qvm-rpc.sls to unblock: Removed (FeaturesRequest, Filecopy, OpenInVM, OpenURL, StartApp, VMRootShell, VMShell). Those files do not exist in /etc/qubes-rpc/policy/, so blockreplace fails (It doesn't look like blockreplace supports creating files if they don't exist 1, so to quickly unblock I just created them manually from scratch and populated the OpenInVM and Filecopy with the expected contents.

After fixing those two things, I am pleased the report that the install completed successfully, and it appears we can use the F25 repo in the : yum will use the URL we specify, and our baseurl is set to https://yum-test.securedrop.org/workstation/dom0/f25

Things that work

• All client functionality (with the changes to rpc logic described above): sync, decryption, replies, OpenInDVM
• USB auto-attach, export and printing
• The templates and grsecurity kernel
• Logs are forwarded to sd-log
• The desktop icon

Thinks that don't work

• The preflight update: PyQt4 is no longer present in dom0, so it does not start
• The RPM package for the config does not install when installed manually on the commandline:

ERROR
  Problem: conflicting requests
    - nothing provides python (abi) = 3.5 needed by securedrop-workstation-dom0-config-0.4.0-1.fc25.noarch

Other comments

• It appears that booting VMs is much slower than in the Qubes 4.0 series
• Qubes 4.1 now has an (opt-in) single policy format that can be used in /etc/qubes/policy.d. Existing RPC definitions still work
• Debug message for gpg in dom0:

gpg: DBG: FIXME: merging secret key blocks is not anymore available
gpg: DBG: FIXME: No way to print secret key packets here

• Good news, make test reports a couple of (expected) failures:
  • fedora-31 template being present used (expected)
  • RPC policies (expected)
  • GPG error checking for the submission key (the key is present and decrypts files) (not expected, perhaps related to debug message above)
  • (EDIT 2020-09-09) popup when restarting whonix with the updater, see: Add support for PyQt5 #610 (review)

[rmol]

I tried a fresh Qubes 4.1 install and hit a number of issues unrelated to our client:

• The root filesystem was created with just 20G, so restoring a VM from the 4.0.3 backup which required more temporary space than that failed, and left the root filesystem full.
• Under XFCE, I ran into enormous .xsession-errors.old QubesOS/qubes-issues#5979, which combined with the 20G root filesystem, quickly resulted in an unusable system as .xsession-errors.old grew out of control.
• Restored standalone VMs based on 4.0.3 debian-10 were basically unusable, in that the window of any graphical application would be mostly black with colorful, dotted vertical lines spaced across it. You could interact with it, but there was no visual feedback. The only way to use the VM was by running a console. Creating a new app VM and copying the private storage over to it got them working again.

[eloquence]

Thanks for these report-backs! :) That concludes our sprint commitments re: 4.1; moving this card to the backlog again for now until we pick up additional 4.1 work.

[eloquence]

@emkll reported a Whonix issue with preflight updates in #610 (review) (also included in summary above):

[eloquence]

Discussed in backlog review today. Given largely positive early reports, the plan of record is to wait until the first RC1 of 4.1 before taking another pass at identifying remaining issues. Keeping on board but moving to bottom of backlog for now. Before then, we'll also want to re-test 4.0.4 final, once that's out (#640).

[eloquence]

On the sprint for tracking purposes; I encourage folks to watch the Qubes 4.1 presentation from the Qubes mini-summit, as we get closer to release, it may also make sense to request a direct chat with Marek & team per Kushal's suggestion. :)

[eloquence]

Still no RC for us to test with; the Qubes mini-summit included a session with highlights from 4.1.

[eloquence]

RC1 is now out. 🎉 We have some time to get our ducks in a row before we can expect a final release: "[W]e expect that it may be anywhere from a few weeks to a few months before we announce the second release candidate."

[eaon]

RC2 was released almost exactly a month ago and over the past week I got to see what happens if one tries to install SecureDrop Workstation on a relatively fresh system. Just sharing some preliminary findings for now, but I will definitely poke at this more in the future.

• I followed the README and used the yum-testing repo to install the dom0 config - that by itself seems to still work fine.
• The first real issues arise during sdw-admin --apply. Qubes 4.1 changes the way templates are managed, and introduced a dedicated tool separate from qubes-dom0-update, which now redirects to qvm-template if the file name implies that what you're attempting to install is a Qubes template. That also affects the attempt to install qubes-template-securedrop-workstation-buster, which is not found because the new tool uses dedicated template repos.
• qvm-template acts more or less like dnf but is its own thing. As opposed to qubes-dom0-update it doesn't open a terminal window in a VM, but the download can be observed in the dom0 terminal. The templates are still .rpm files, and the repositories are RPM repositories. I created a temporary one with just the SDW templates to see if that would work.
• The repo definitions are loaded from /etc/qubes/repo-templates/qubes-templates.repo, qvm-template ignores any other .repo file in that folder at the moment, and the 4.1-rc2 version of the tool has a bug that breaks the --repo-files argument (might already be fixed upstream, I've only tested what's shipped with rc2), so I just appended my test repo to the file. With that securedrop-workstation-buster installs fine when using qvm-template directly.
• The salt states referencing installed template packages still fail however - even though the template was installed, the system package manager isn't aware of the rpm associated with it. qvm.template_installed is available though (and also works). Being new to SaltStack, I'm not sure how or if that can be made to interact with the require: and watch: bits as seen in sd-sys-vm.sls (or if that's even necessary anymore).

I haven't resolved all the template install/check related issues in the salt states yet, but even without that it's already noticable that another hiccup relates to RPC policies (as mentioned in the original issue). SDW expects some files to be in /etc/qubes-rpc/policy/ but the policy format has changed and and some files moved to /etc/qubes/policy.d/. When speaking to @eloquence about this earlier today it was suggested that updating them to the new 4.1 format might be relatively straightforward, so that's something I plan to take a stab at the next time I'm returning to this 🐰 🕳️

[gonzalo-bulnes]

@eaon Regarding the RPC policies, my understanding is that we don't have to modify the existing policies yet, as R4.1 policies system remains backwards-compatible with R4.0 - source.

That being said, I couldn't confirm that on my R4.1 and am waiting for fresh ideas to pursue the troubleshooting further. (I asked the question on the Qubes OS forum.)

[eloquence]

For the current sprint, @rocodes, @eaon @conorsch and @creviera will collaborate on a time-boxed (1-2 day) spike with the goal to get the SecureDrop Workstation to be fully installable on 4.1 RC (but no conditional logic to make it also installable on 4.0.4 using the same codebase).

[conorsch]

Tried a clean install of 4.1-rc3, and verified what's already in the reports above. @eaon since you mentioned you're working on some conditional logic for the qvm-template change, I'd love to peek at a branch if you've got one already. Took some screenshots for my own reference on other issues; the gpg key verification task, mentioned in #600 (comment), looks like:

We'll have to come up with a smarter command to verify the pubkey is valid; probably adding some tempdir keyring import action to the associated check.

During RPM download, the flow is slightly different from what we've documented:

Crucially, the dnf download <package> task does not verify signatures, so we must continue to use the manual verification command. That was surprising to me, given the _pkgverify macro changes in 4.1, but those must only take effect on installation, not download (the opposite of apt in that respect). Also submitted a small docs PR that doesn't need to wait for 4.1: freedomofpress/securedrop-workstation-docs#98

[eaon]

sys-usb being disposable by default in 4.1 was mentioned in #751, but decided to be out of scope for that PR, so bringing it up here:

AFAICT, the easiest way to handle disposable sys-usb is by cloning fedora-34-dvm as (for example) sd-fedora-34-dvm, modifying that disposable template with our udev rules and scripts, and then switching out the template that sys-usb uses. I tried this manually and it works 🎉 but I'm wondering if there's any downsides to this approach that I'm not aware of? Otherwise I'd be happy to cast this into salt states relatively soon

[sssoleileraaa]

As you can see above, we are linking remaining work to this tracking issue with a focus on getting to the first phase of being able to test SDW nightlies on Qubes 4.1.

By tomorrow, all SDW packages should be on apt-test, but we still need a way to switch over to nightlies which is in development (close to ready for review) in #762.

The biggest blocker, as I see it, for not being able to test nightlies next week is freedomofpress/qubes-template-securedrop-workstation#25, which might only be happening for me (needs a repro) but it's preventing me from publishing a new template package to yum-test.

[eloquence]

Today, I re-provisioned current main (with the changes in #784) against my long-running prod instances. I was able to observe the same issue @eaon observed during his QA run: I'm unable to log into the SecureDrop Client. The call_api calls fail instantly with a "No response from proxy" error. I did not observe AppArmor or RPC denials related to that, but will take another close look later.

I was able to install the SecureDrop Client in sd-app from https://github.com/freedomofpress/securedrop-client/ using the python -m securedrop_client method documented in the README (from "To run a different version of the client, ...") and then run it. [*] Logging in initially timed out, but after restarting all VMs, I was able to log into my server reliably, download files, delete sources, and open files in disposable VMs. 🎉

What's the difference? Unclear as of yet. As I understand it, sd-proxy is used in the same manner, regardless of how the client is invoked.

I tried again running via securedrop-client, and again hit the proxy error. I was able to open the client in offline mode. When attempting to open a file, however, I hit a new AppArmor denial, so after the sd-proxy issue is overcome, I expect we have some more work to do there.

[*] I had to manually pip install PyQt5 and pyqt5-sip; as I understand from @gonzalo-bulnes, once freedomofpress/securedrop-client#1496 is merged, this should no longer be necessary.

[eloquence]

The proxy errors were indeed the result of another AppArmor policy issue. Note timing on the logs below (visible both via tailing /var/log/syslog or via sudo journalctl -f):

Jun 27 19:27:41 localhost 2022-06-27 19:27:41,099 - securedrop_client.logic:126(call_api) ERROR: 'No response from proxy'
Jun 27 19:27:41 localhost 2022-06-27 19:27:41,100 - securedrop_client.logic:554(completed_api_call) DEBUG: Completed API call. Cleaning up and running callback.
Jun 27 19:27:41 localhost kernel: [ 6263.552579] audit: type=1400 audit(1656383261.096:22): apparmor="DENIED" operation="open" profile="/usr/bin/securedrop-client" name="/dev/xen/hypercall" pid=4380 comm="qrexec-client-v" requested_mask="wr" denied_mask="wr" fsuid=1000 ouid=0

The call here is to qrexec-client-vm; it's abbreviated in the AppArmor log to qrexec-client-v. This is the qrexec tool that's used by the SecureDrop SDK (the library used by the SecureDrop Client to perform API requests) to send commands to the proxy:
https://github.com/freedomofpress/securedrop-sdk/blob/0909cb9a20c791b43d10c1d473cbe4dcd4c32be8/sdclientapi/__init__.py#L53-L58

We've already allow-listed other /dev/xen devices; @marmarek's commit message in QubesOS/qubes-linux-utils@4fe08d3 suggests that Linux kernel versions before 4.18 did not make this device node available to Xen, so on our older 4.14 kernel we shipped with the Buster template, we would not have hit this issue on either 4.0 or 4.1.

On the newer kernel we now ship with the Bullseye template, the device node is available, Xen tries to use it, and it errors out. This also causes disposable VM operations to error out if you do get into the client, as those also rely on qrexec-client-vm.

Once the requisite permission is added, both logging in and opening files works as expected. 🎉 I'll open an AppArmor PR to resolve.

[eloquence]

I've not stepped through the formal acceptance tests yet, but a quick feature tour looks good:

• syncing and decrypting work without issues
• different file types (tested JPG, M4A, DOC, PDF, PPT) open in disposable VMs as expected
• device auto-attachment and export to USB worked flawlessly (!) - this is with the "sys-usb is disposable" 4.1 default we'll recommend to users
• deletion (source or files/message) works without problems
• sending replies works without problems
• starring, "download all" works without problems

More testing to come :)

[eloquence]

First phase of acceptance testing all good. 🎉

Environment: Qubes 4.1 fresh install with disposable VM setup for sys-usb/sys-firewall, running against SD 2.4.0 prod server on hardware

SecureDrop Workstation test scenarios

Verify tor connection to Journalist API

• Create VM for accessing JI via Tor Browser: qvm-create --template whonix-ws-16 --property netvm=sd-whonix --label orange sd-research. Open Tor Browser in that VM and confirm you can log in to the Journalist Interface. This confirms that sd-whonix is configured correctly (but does not use sd-proxy).
• Change the netvm to sys-whonix and confirm you can load the public Source Interface, but not the Journalist Interface. (N.B. you must leave the netvm set to sys-whonix, otherwise make clean and sdw-admin --uninstall will fail.)

Verify default DispVM settings

• Open a shell in a non-SDW VM, e.g. sd-dev. Download a PDF file and open it via: qvm-open-in-dvm <pdf_file>. Confirm it opens in a DispVM, and that the DispVM is based on sd-viewer.
• Open a shell in sd-app and find an already downloaded submission in ~/.securedrop_client/data/. Run xdg-open <file_path> and confirm it opens in a DispVM, and that the DispVM is based on sd-viewer.

RPC Policies

• Open a shell in a non-SDW VM, e.g. sd-dev. Run: QUBES_GPG_DOMAIN=sd-gpg qubes-gpg-client -k. Confirm that the request is denied, i.e. you do NOT see pubkey info for the SecureDrop Submission Key.
• Try to copy/paste from the Client to a non-SDW VM, e.g. sd-dev. Confirm you cannot.
• Add the clipboard tags to sd-dev as described in the docs, and repeat the copy/paste procedure. Confirm it works.

Logging

• Logs are sent to sd-log VM
• No warnings are displayed about logging denials (e.g. sd-log to sd-log; see sd-log tries to log to itself, but loopback qrexec connections aren't supported #755)
  • Note to testers: It is expected that sd-gpg does not send logs to sd-log. If you do see new logs from sd-gpg, that would be a regression.

GUI updater

• Reboot the workstation after installing SDW. Confirm that the prelaunch updater window appears automatically after logging, prompting for an update.
• Proceed with GUI updater, confirm it runs without errors.

[eloquence]

Still all good! 🎉 Skipped ahead to ensure I test export fully today, no issues there. Tomorrow I'll prioritize file type support as another high risk area.

Environment: Qubes 4.1 fresh install with disposable VM setup for sys-usb/sys-firewall, running against SD 2.4.0 prod server on hardware

Login

• when SecureDrop desktop icon is double-clicked, preflight updater is displayed
• After preflight updater runs, when user clicks Continue, login dialog is displayed
• In login dialog:
  • show/hide password functionality works
  • incorrect password cannot log in
  • 2FA token reuse cannot log in after password failure
  • invalid 2FA token cannot log in
  • valid credentials and 2FA can log in

Sources

• after valid login:
  • the login dialog closes
  • source data is downloaded and source list is populated
  • user is prompted for GPG key access
  • submissions and replies are decrypted
  • the source list is displayed but no sources are selected by default
  • the conversation view is not populated
• when a source is selected in source list:
  • conversation view is populated with source conversation
  • a source message containing HTML is displayed as unformatted text
  • source submissions have an active Download button
  • source submission compressed file size is displayed accurately

(Skipping ahead to export)

Export

• When Export is first clicked on a submission:
  • the "Preparing to export..." message is displayed
  • the sd-devices VM is started
  • the user is prompted to insert an Export USB
  • On clicking Cancel, the prompt closes and the file is not exported
• When Export is clicked on the submission again:
  • the "Preparing to export..." message is displayed
  • the user is prompted to insert an Export USB
  • When the user inserts an invalid Export USB, attaches it to the sd-devices VM and clicks OK:
    • a message is displayed indicating that the Export USB is invalid and
      the user is prompted to insert a valid device
• When Export is clicked on the submission again:
  • the "Preparing to export..." message is displayed
  • the user is prompted to insert an Export USB
  • When the user inserts a valid Export USB, attaches it to the sd-devices VM, and clicks OK:
    • the user is prompted for the Export USB's password
  • When the user enters an invalid Export USB password and clicks Submit:
    • a failure message is displayed and the user is prompted to enter the password again
  • When the user enters an valid Export USB password and clicks Submit:
    • the file is saved to the Export USB
• When the user detaches the Export USB and mounts it on another VM or computer:
  • the decrypted submission is available in on the Export USB, in a directory sd-export-<timestamp>/export_data

[eaon]

Started acceptance testing with non-merged printing patches also looking quite good, but I'm about to wipe my test machine to start from zero because of a couple of reports of problems with the install process on fresh machines. Here's how far I got:

Verify tor connection to Journalist API

• Create VM for accessing JI via Tor Browser: qvm-create --template whonix-ws-16 --property netvm=sd-whonix --label orange sd-research. Open Tor Browser in that VM and confirm you can log in to the Journalist Interface. This confirms that sd-whonix is configured correctly (but does not use sd-proxy).
• Change the netvm to sys-whonix and confirm you can load the public Source Interface, but not the Journalist Interface. (N.B. you must leave the netvm set to sys-whonix, otherwise make clean and sdw-admin --uninstall will fail.)

Verify default DispVM settings

• Open a shell in a non-SDW VM, e.g. sd-dev. Download a PDF file and open it via: qvm-open-in-dvm <pdf_file>. Confirm it opens in a DispVM, and that the DispVM is based on sd-viewer.
• Open a shell in sd-app and find an already downloaded submission in ~/.securedrop_client/data/. Run xdg-open <file_path> and confirm it opens in a DispVM, and that the DispVM is based on sd-viewer.

RPC Policies

• Open a shell in a non-SDW VM, e.g. sd-dev. Run: QUBES_GPG_DOMAIN=sd-gpg qubes-gpg-client -k. Confirm that the request is denied, i.e. you do NOT see pubkey info for the SecureDrop Submission Key.
• Try to copy/paste from the Client to a non-SDW VM, e.g. sd-dev. Confirm you cannot.
• Add the clipboard tags to sd-dev as described in the docs, and repeat the copy/paste procedure. Confirm it works.

Logging

• Logs are sent to sd-log VM
• No warnings are displayed about logging denials (e.g. sd-log to sd-log; see sd-log tries to log to itself, but loopback qrexec connections aren't supported #755)

GUI updater

• Reboot the workstation after installing SDW. Confirm that the prelaunch updater window appears automatically after logging, prompting for an update.
• Proceed with GUI updater, confirm it runs without errors.

Client scenarios

Scenario: Online mode

Login

• when SecureDrop desktop icon is double-clicked, preflight updater is displayed
• After preflight updater runs, when user clicks Continue, login dialog is displayed
• In login dialog:
  • show/hide password functionality works
  • incorrect password cannot log in
  • 2FA token reuse cannot log in after password failure
  • invalid 2FA token cannot log in
  • valid credentials and 2FA can log in

Sources

• after valid login:
  • the login dialog closes
  • source data is downloaded and source list is populated
  • user is prompted for GPG key access
  • submissions and replies are decrypted
  • the source list is displayed but no sources are selected by default
  • the conversation view is not populated
• when a source is selected in source list:
  • conversation view is populated with source conversation
  • a source message containing HTML is displayed as unformatted text
  • source submissions have an active Download button
  • source submission compressed file size is displayed accurately
• when the upper right 3-dot button is clicked:
  • a menu is displayed with a delete source account option
  • when delete source account is selected:
    • the source is deleted from the source list and the conversation view is blanked
    • the source is deleted from the server and not restored on next sync
    • source submissions and messages are removed from the client's data directory
• when a source is starred in source list, and the client is closed and reopened in Online mode:
  • the source is still starred in the source list

[eloquence]

I've uploaded all our test files to my instance and tested opening them in disposable VM. That covers: JPG, MP3, ODP, ODS, OGG, PNG, PPT, SVG, TIFF, WAV, WEBP, XLSX, DOC, DOCX, ODT, RTF, PDF, M4A, PPTX, and ZIP (including the ZIP's contents). No issues -- all opened in the expected viewer/player applications in disposable VMs. (#588 is still an issue for files opened with LibreOffice, unfortunately.)

[eloquence]

Just a quick note that dom0 tests are still all passing for me ✔️, with the exception of the known securedrop-grsec issue that will be resolved once we use the package version and dependencies from main on apt-test (cf. #793 and freedomofpress/securedrop-builder#354). That issue, in any event, is a test-only issue and does not cause update failures (I've been able to run the updater daily without errors on 4.1, with the exception of one Whonix timeout).

[eloquence]

A couple of additional checks not covered in our test plan:

• update-xfce-settings script still works as expected to disable suspend-to-disk and hibernate (which are unsafe in combination with FDE) and to make desktop icons more legible
• Exporting logs to a target VM as documented still works

[eloquence]

Did a fresh install (SDW only on existing 4.1 install) with all recent changes (#802, #795, latest nightlies with securedrop-export changes) and all looks good -- fedora-36 was installed as expected, CUPS was enabled as expected, and export still works fine as expected. :)

[eloquence]

Environment:

• Qubes 4.1 (newly installed, but has since been through a few SDW provisioning cycles)
• prod environment install
• Prod server running SD 2.4.0
• SDW template was removed prior to installation

Abbreviated test:

• All SecureDrop packages at released evrsions
• Updater completes without errors
• SDW VMs are running the expected 5.15.41 grsec kernel
• Logging into the Client and syncing works
• Downloading files works
• Opening files in a subset of supported formats in disposable VMs works
• Deleting sources works
• Sending a reply works
• Exporting files works
• Tagging for copy/paste works
• Printing with wrong USB plugged in shows expected "Attach printer" error message
• journalctl -f in dom0 while running the app shows no denials/errors
• Logs are aggregated to sd-log; sd-app logs show no denials/errors
• Lid close = shutdown

[eloquence]

Completed another production install on 4.1.0 (including OS reinstall) as part of verifying docs changes in freedomofpress/securedrop-workstation-docs#112 and did some quick smoke testing (login, disp VMs, export, kernel version, etc.), all good. :)

[sssoleileraaa]

We finished the release and related scheduling last week, and the docs were published today, so this issue can be closed. Nice work everyone!