Remove buster, add failing bookworm jobs

[legoktm]

Note, #372 must be merged before this one!

We'd like bookworm jobs to be able to fail, but when they do fail it
should make an appropriate level of noise so we can investigate why they
failed. But they should also not stop the bullseye packages from being
uploaded!

So we split the bookworm jobs into a separate track ("build2"), and have
separate push jobs for each version. The push-bullseye job goes before
the push-bookworm one in case bookworm jobs are failing.

Note that currently all the bookworm jobs are failing because of
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017046. I've put in a
fix for that, it should be in bookworm in the next 3-5 days.

Refs #369.

[legoktm]

Here's what the nightly job ordering looks like:

[eloquence]

What should the policy on bookworm checks be? Should it be acceptable to merge PRs even if bookworm checks are failing?

If the answer is yes, just noting that, as I understand it, those commits will show up with a red "X" in the commit history on GitHub, regardless of whether the check is required or not. Not sure if there's a good way around that.

[legoktm]

What should the policy on bookworm checks be? Should it be acceptable to merge PRs even if bookworm checks are failing?

Yes. I think people should quickly check to see if their change is causing the failure, and if so fix it (or less preferred, log an issue).

If the answer is yes, just noting that, as I understand it, those commits will show up with a red "X" in the commit history on GitHub, regardless of whether the check is required or not. Not sure if there's a good way around that.

Yeah :-(

[legoktm]

Hrmph.

# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux bookworm/sid"
NAME="Debian GNU/Linux"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

No $VERSION_CODENAME which we use in quite a few places... 🤔

This was already filed at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008735

[legoktm]

Remaining bookworm failures:

• securedrop-client: needs some wheels to be built
• securedrop-proxy: same
• securedrop-workstation-grsec: I didn't copy over the changelog, leaving intentionally broken since it's going to go away soon I hope.

[legoktm]

I'm going to leave removing the Python 3.7 wheels for another PR just so the resigning of the checksums file doesn't cause issues on rebase.

[eloquence]

securedrop-client: needs some wheels to be built
securedrop-proxy: same

Do you want to do that as part of this PR before merging?

[legoktm]

No strong preference on my end, I think if I get to it before this is reviewed/merged I can add it to this PR, otherwise it can go separately.

[gonzalo-bulnes]

I'm realizing how unfamiliar I am with this repo 😝 so I'll only comment.

• The goal is clear to me ✅
• The limitations in what we can do are too ✅
• And with that in mind, the mechanics are clear too, with caveats that I think are acceptable (We'll be merging red builds somewhat regularly. Because it's a conscious decision I can live with that :P) ✔️
• I'm not sure who signs the checksums file, so I haven't checked that. 🍊
• I cannot spot any mistakes in the implementation. Take that with a grain of salt, but I also see that CI looks as expected, so I would go forward and ship it. ✔️

I'll let you give your thoughts too @eloquence !

[legoktm]

• I'm not sure who signs the checksums file, so I haven't checked that. 🍊

Anyone who has a key in the pubkeys/ folder can resign the file.

[cfm]

I love how this change shows off #371's parameterization of this CI workflow!

• Prerequisite Use plain Debian images in CI #372 is merged
• CI is passing:
  • bullseye
  • bookworm, except for expected failures:
    • securedrop-client itself does not yet have a pinned hash for Python 3.10 wheels (pending Add sha256sums of rebuilt wheels on Python 3.10 (bookworm) securedrop-client#1554)
    • securedrop-proxy itself does not yet have a pinned hash for Python 3.10 wheels (pending Add sha256sum of rebuilt PyYAML wheel on Python 3.10 (bookworm) securedrop-proxy#104)
    • securedrop-workstation-grsec does not (and will not) have its own changelog-bookworm
• Checksums are signed:

  user@sd-dev:~/securedrop-debian-packaging$ gpg --verify bootstrap-sha256sums.txt.asc 2>&1 | head -n 5
  gpg: assuming signed data in 'bootstrap-sha256sums.txt'
  gpg: Signature made Wed 31 Aug 2022 12:27:41 AM PDT
  gpg:                using RSA key D8CB59F05DBB9E0538C4819DF105F8101B05269B
  gpg: Good signature from "Kunal Mehta <[email protected]>" [unknown]
  gpg:                 aka "Kunal Mehta <[email protected]>" [unknown]
  user@sd-dev:~/securedrop-debian-packaging$ gpg --verify sha256sums.txt.asc 2>&1 | head -n 5
  gpg: assuming signed data in 'sha256sums.txt'
  gpg: Signature made Thu 01 Sep 2022 03:46:14 PM PDT
  gpg:                using RSA key D8CB59F05DBB9E0538C4819DF105F8101B05269B
  gpg: Good signature from "Kunal Mehta <[email protected]>" [unknown]
  gpg:                 aka "Kunal Mehta <[email protected]>" [unknown]

In Monday's Workstation hangout we discussed #373 (comment) and agreed on a criterion for assessing provisional bookworm (and future Debian testing) builds (my summary notes):

If it's your change that's causing the bookworm job to fail, you should fix it before merge. If it's upstream, it's a judgment call.

As always, I would prefer to record this guidance to maintainers explicitly somewhere. @legoktm, do you think could be done usefully in a pull-request template here? or will that be necessary per upstream component repository (securedrop-client, etc.)?

[legoktm]

As always, I would prefer to record this guidance to maintainers explicitly somewhere. @legoktm, do you think could be done usefully in a pull-request template here? or will that be necessary per upstream component repository (securedrop-client, etc.)?

Yes to both, in that bookworm job failures might need to be ignored here as well as in component repositories. I note that we currently don't even have a pull request template here :<, and the README is already a mess of documentation. Since it's a general policy that will span multiple repos, maybe it belongs better in the new developer docs?

[eloquence]

My suggestion would be for a short paragraph into a PR template for now:

== Checklist ==

[ ] I've verified that this PR does not introduce any new bookworm test failures

Or similar.

[legoktm]

OK, how's that?

[cfm]

fbceaa8 nicely captures the spirit of "bookworm or bust". :-) Thanks, @eloquence and @legoktm! Approving and merging on the basis of #373 (review).

[legoktm]

Wheee, thank you to all 3 reviewers :-)