QA and release 0.5.0

[apyrgio]

To ensure that new releases do not introduce regressions, and support existing and newer platforms, we have to run some QA tests. Copied from RELEASE.md:

• In .circleci/config.yml, add new platforms and remove obsolete platforms
• Bump the Python dependencies using poetry lock
• Make sure that the tip of the main branch passes the CI tests.
• Make sure that the Apple account has a valid application password and has
  agreed to the latest Apple terms (see macOS release
  section).
• Create a test build in Windows and make sure it works:
  • Check if the suggested Python version is still supported.
  • Create a new development environment with Poetry.
  • Build the container image and ensure the development environment uses
    the new image.
  • Run the Dangerzone tests.
  • Build and run the Dangerzone .exe
  • Test some QA scenarios (see Scenarios below).
• Create a test build in macOS (Intel CPU) and make sure it works:
  • Check if the suggested Python version is still supported.
  • Create a new development environment with Poetry.
  • Build the container image and ensure the development environment uses
    the new image.
  • Run the Dangerzone tests.
  • Create and run an app bundle.
  • Test some QA scenarios (see Scenarios below).
• Create a test build in macOS (M1/2 CPU) and make sure it works:
  • Check if the suggested Python version is still supported.
  • Create a new development environment with Poetry.
  • Build the container image and ensure the development environment uses
    the new image.
  • Run the Dangerzone tests.
  • Create and run an app bundle.
  • Test some QA scenarios (see Scenarios below).
• Create a test build in the most recent Ubuntu LTS platform (Ubuntu 22.04
  as of writing this) and make sure it works:
  • Create a new development environment with Poetry.
  • Build the container image and ensure the development environment uses
    the new image.
  • Run the Dangerzone tests.
  • Create a .deb package and install it system-wide.
  • Test some QA scenarios (see Scenarios below).
• Create a test build in the most recent Fedora platform (Fedora 38 as of
  writing this) and make sure it works:
  • Create a new development environment with Poetry.
  • Build the container image and ensure the development environment uses
    the new image.
  • Run the Dangerzone tests.
  • Create an .rpm package and install it system-wide.
  • Test some QA scenarios (see Scenarios below).
• Create a test build in the most recent Qubes Fedora template (Fedora 38 as
  of writing this) and make sure it works:
  • Create a new development environment with Poetry.
  • Run the Dangerzone tests.
  • Create a Qubes .rpm package and install it system-wide.
  • Test some QA scenarios (see Scenarios below) and make sure
    they spawn disposable qubes.

(see also the previous QA for 0.4.2)

[eloquence]

I've used qa.py on an Ubuntu 23.04 host system to step through the automated and manual tests for the Ubuntu 22.04 target platform.

I was not able to use qa.py on an Ubuntu 20.04 host system. The process errored out with "Error: kernel does not support overlay fs: unable to create kernel-style whiteout: operation not permitted". fuse-overlayfs is installed. I didn't dig further into this and unless you have some low-hanging fruit things to test it's probably not worth doing so since 20.04 is pretty long in the tooth.

Automated tests

Automated test results all look good. I did get some deprecation warnings for which I've opened #595. There's also some warning output related to code coverage reporting and shibokensupport. The full output can be seen here: https://gist.github.com/eloquence/4a382d55ccb286e83de83043e88173bf

Manual tests

Dangerzone retains the settings of previous runs

The "Check for updates" setting does not persist for me. I've not tested if this problem occurs outside the QA environment. I've opened #596 for this.

Other settings, e.g. "Open after converting", do persist.

Updating Dangerzone handles external state correctly

I've not tested this yet. @deeplow I'd appreciate some pointers on this one - is this also testable with the qa.py setup, or should I manually build Dangerzone on the host system? If it can be done using qa.py, could you give me the steps I need to go through to ensure I'm testing both the container image and the Debian package in the expected manner?

All other test scenarios LGTM.

[deeplow]

Updating Dangerzone handles external state correctly

I've not tested this yet. @deeplow I'd appreciate some pointers on this one - is this also testable with the qa.py setup, or should I manually build Dangerzone on the host system? If it can be done using qa.py, could you give me the steps I need to go through to ensure I'm testing both the container image and the Debian package in the expected manner?

This is not trivial to test and it's particularly hard using our QA environment. Personally, I haven't been able to test that for the last two releases. But this is not ideal, so I've opened an issue to streamline this process #597

[deeplow]

I also noticed that it was asking to test scenario 9 (Qubes-specific), so I marked it as "skippable" on Linux platforms #598

[deeplow]

Fedora 38 QA

I have just done the steps for QA on Fedora 38.

Dangerzone retains the settings of previous runs

The "Check for updates" setting does not persist for me. I've not tested if this problem occurs outside the QA environment. I've opened #596 for this.

Other settings, e.g. "Open after converting", do persist.

I can confirm that #596 also affects Fedora 38.

[eloquence]

I was also able to test the upgrade scenario on Ubuntu 22.04 thanks to #605, no issues to report. :) Haven't gotten to Qubes yet. I should have some time for exploratory testing Monday afternoon. Please don't treat that as a blocker for the release, though, I know you've tested this extensively already.

Once we land Ubuntu 23.10 support I think we're otherwise looking in good shape - please correct me if I'm missing anything. The release mechanics themselves take a good amount of time, so I suggest we touch base Monday to settle on exact time/date. We may also want to confirm Lee's availability in case we hit any issues that require infra support.

[deeplow]

All looks good on the large tests: 80 failures / 9748 docs.