[QA] The apt-test HTTPS cert fails on older versions of Focal

[conorsch]

Description

Older versions of Focal installing packages from https://apt-test.freedom.press fail during apt-get update, with this error:

Steps to Reproduce

Use this dockerfile to observe the issue.

Dockerfile

# Intentionally using an old tag, to test old Focal packages
# See here for options: https://quay.io/repository/freedomofpress/sd-docker-builder-focal?tab=tags
# See related expiry info for LE HTTPS certs here:
# https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
FROM quay.io/freedomofpress/sd-docker-builder-focal:2021_04_14

RUN apt-get update && apt-get install -y curl apt-transport-https gnupg2

# Uncomment the upgrade task below to see a working build!
# RUN apt-get upgrade -y

RUN echo "deb [arch=amd64] https://apt-test.freedom.press focal main" | tee /etc/apt/sources.list.d/fpf-apt-test-repo.list

RUN curl -s https://raw.githubusercontent.com/freedomofpress/securedrop/develop/install_files/ansible-base/roles/install-fpf-repo/files/apt-test-signing-key.pub | apt-key add -

# The following line fails, with:
#
#   9 0.711 Err:4 https://apt-test.freedom.press focal Release
#   9 0.711   Certificate verification failed: The certificate is NOT trusted.
#   The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification. [IP: 206.81.1.102 443]
#
RUN apt-get update

Expected Behavior

Repo is configured, apt lists are updated, packages are installed just fine.

Actual Behavior

After repo is configured, apt list update fails, with:

Certificate verification failed: The certificate is NOT trusted.
The certificate chain uses expired certificate.  
Could not handshake: Error in the certificate verification. [IP: 206.81.1.102 443]

Comments

We believe the LE root cert transition is affecting us here https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ @zenmonkeykstop first observed this issue over the weekend, motivating #6116 & #6117, and is now working on a fix that'll likely make it into 2.1.0~rc2 (#6103).

[gonzalo-bulnes]

For what is worth: Similarly to #6103 (comment), I encountered the error apt cache update failed in the install-fpf-repo task, when attempting a fresh install of SecureDrop 2.0.2 (git checkout 2.0.2 && ./securedrop-admin install) on Ubuntu 20.04 servers that were not fully up-to-date.

Upgrading them (sudo apt update && sudo apt upgrade) got me past that step, and I notice that the ca-certificates was upgraded. That can be a way forward / detail to keep in mind until #6120 is part of the current release.

[conorsch]

Thank you, @gonzalo-bulnes! Were these VMs you were installing on? My recollection is that the Ubuntu ISO installer, when installing on hardware, will automatically pull in the latest packages, sidestepping this problem for fresh installs. Either way, the fix mentioned above will be released as part of v2.1.0 next week, and is already present on the develop branch (but not the previous signed release, 2.0.2, which you used).

[conorsch]

Pleased to report that while testing 2.1.0~rc2 in #6103, this issue did not occur. To test, I used prod VMs based on 202107.28.0 Focal (which failed previously, in the manner described in the OP), did not upgrade them manually after creation, and simply ran ./securedrop-admin install on the 2.1.0-rc2 tag. Worked well.

[gonzalo-bulnes]

@conorsch These were physical machines, but I had performed the Ubuntu installation a while ago. I took a note ☝️ that a reminder to upgrade Ubuntu before starting the installation my be worth adding to the docs.

[conorsch]

Ah, thank you for clarifying! As of 2.1.0 final (slated for next week), the upgrade-everything logic will be automatic. So I don't see a pressing need for a docs change. But please let me know if you disagree!