Tech Meeting Notes 2020 06 04

SecureDrop Tech Meeting, June 4, 2020

Topic: Migration to a new server OS before the Ubuntu 16.04 end-of-life

16.04 reaches EOL 2021-04-30

(Kushal) Even Ubuntu team doesn't do sufficient testing of its upgrades
(Conor) We experimented with using Journalist Interface for notifying admins, which proved very valuable
(Conor) Went on-site a few times, which helped, but we may not be able to do it this time
(Mickael) There were package differences between versions. Some versions of Ubuntu no longer have certain packages. Some Python libraries interacted differently with the system.
(Kushal) Our use of dh-virtualenv eases the burden for future upgrades
Ultimately lost only a few instances, and (anecdotally) a decent number did it without interactive assistance
(Kevin) We have more interaction with folks now, which should be a big help

(Kushal) Ubuntu 20.04, which will be supported until 2025
(Kev) If we can do unattended upgrade to 18.04, I would prefer it
(Conor) More research required, but quite skeptical. Also feel that we should aim for 20.04
(Kev) Agree, if manual intervention required, go straight to 20.04
(Mickael) Do we expect issues with Python support/versions?
(Kushal) We'll be upgraded to the latest Python
(Conor) 3.8 on 20.04 I think
(Conor) Getting us to 20.04 will give us breathing room for investigating other server OS options

CONSENSUS:

Additional research required to understand whether or not unattended upgrade is at all feasible
If possible, upgrade to 20.04 seems highly desirable to give ourselves breathing room

(Kushal) Short window during which both operating systems are supported
(Conor) Practically speaking we have to support 16.04 until April 30.
(Conor) Would like to aim for 3 months window
(Kev) This means we're on the hook for testing every release on both operating systems

CONSENSUS:

(Kev) Need to understand time and complexity involved. do-release-upgrade was less work and we were confident that it was not going to be error-prone, but doubling that up will increase wall time and points of failure.
(Erik) Worried about % of instances dropping off if we force reinstall
(Conor) As a maintainer I do like the "fresh start" but agree it's a barrier
(Mickael) Have observed configuration drift when you run upgrades, also an important area to research

CONSENSUS:

(Kev) Would increase complexity for admins -- will have to work with journalist teams/web teams
(Conor) We can start advocating move to v3 independently
(Kev) Are there security concerns re: v2?
(Conor) Yes, but should do fresh write-up
(Ro) We've made a little push there, but we don't really get a response
(Conor) Crypto is much stronger - v2 uses crypto that is not state of the art both in terms of key length & primitives. One of the best changes we can make for source protection.
(Ro) Perhaps v2 EOL could be 2-3 months later
(Mickael) There are also implications for comms with existing sources. Note that with HTTPSE rulesets, onion address can be replaced transparently, which could reduce comms burden.
(Conor) Important to get folks to do Tails upgrades for v3 migration
(Kev) We could push for two major updates, end-of-life for v2 services(John) seconded, because v2+v3 QA was painful, and because as Conor pointed out v3 helps sources -- why wait a year?
(Erik) In favor of early notification in JI, but a single release that deprecates v2 & Ubuntu 16.04
(Ro) Could add notice to Tails graphical updater for deprecation warning

CONSENSUS:

STILL TBD:

An early release that deprecates v2, or remove v2 support + Ubuntu 16.04 support with the same release

(Conor) Two do-release-upgrade runs of a prod install. What's breaking?
(John/Kushal) App will not work at all, but interested in other breakage
(Kushal) Packaging the SecureDrop app on 20.04, determining if it can work
(Conor) Once you have done do-release-upgrade, try to unbreak and see where you get stuck
(Mickael) In terms of configuration drift, we won't be able test Trusty->Xenial->Bionic->Focal path. Need to understand package support status for critical dependencies. I've observed that SSH client config from 16.04 doesn't work well in later releases -- this is the kind of thing we have to guard against.
(Mickael) Conor, do you foresee issues with nested-virt in CI?
(Conor) I expect it'd be more well-maintained than older systems, interested in spike, but after we do do-release-upgrade stuff.
(Mickael) Remember the issue with CPU passthrough option
(Conor) Oh right, let's do a spike to identify bugs early
(John) Should we do spike on clean install?
(Conor) Need to do packages first. That said, not a bad way to get an additional perspective.
(Mickael) Agree, the install process gives us a more structured report-out.
(Erik) Will also give us sanity check on OS-level documentation & steps

CONSENSUS:

(John) Switching to unattended upgrades?
(Conor) Yes, this is a wonderful opportunity to do it -- support a spike around that. Helps us to get to a distro-standard way.
(Kev) Need to ensure that nightly reboot doesn't step on unattended upgrades
(Mickael) unattended upgrades support automatic rebooting https://help.ubuntu.com/community/AutomaticSecurityUpdates
(Conor) Recommend that we eliminate universe channel, but upgrade everything that is allowed on the system. This would allow us to ditch the logic described in https://github.com/freedomofpress/securedrop/issues/3376
(Mickael) Note that this will cause more packages to be upgraded -- not necessarily problem, but may increase the risk of breakage
(Ro) I'd like to figure out what our long term path is for OSSEC

CONSENSUS:

Let's investigate improving channel logic (resolving #3376) / removing universe channel as part of Ubuntu 20.04 transition