Containerized SD server - application update story

[msheiny]

Feature request

Description

This is semi-related to #2545 -- We need a good story on how to keep SecureDrop application containers up-to-date on a CoreOS system. There are some conflicting strategies here that I'm aware of, others that come to mind please chime in. Lets use this ticket to focus discussion.

• Having SecureDrop organization servers be a member of an FPF run kubernetes cluster. This would provide FPF direct control over what SD applications are running against a slew of instances. We could run something like Techtonic (or our own k8s stack) and require all SecureDrop instances to join our cluster in order to get updates. Obviously with Techtonic there is a licensing element we'd need to hammer out -- as well as a slew of security questions that would come up. I'm not certain what we'd tell organizations that want to spin up an instance and not be under our cluster - we'd have to come up with an alternative story for that if we happen to go this route.

• Running our own cron-like systemd service to poll for the latest stable securedrop version regularly. Pulling in all the relevant securedrop containers that are needed. We might want to throw something 9like docker-compose onto the base coreos system](https://medium.com/@wiwatSrt/install-docker-compose-on-coreos-f915793ffc2f). I've also found articles of people installing python and using ansible against the base system. We could possible be polling and downloading ansible-playbooks that will stand-up the containers against the latest securedrop dependencies. Might need to roll something with TUF here? I dunno, I see this going in a lot of potential directions.

User Stories

As a securedrop adminisrtator, I want automatic, regular updates of the SecureDrop code-base.

As a securedrop developer, I want to be able to push inter-dependent containers to securedrop instances securely and with little development effort.

[squeed]

I'm not sure I would recommend Kubernetes for SecureDrop - and I'm a kubernetes contributor. It's way, waaay too large a surface area for a project like this, and it doesn't necessarily solve the specific SecureDrop deployment story much more neatly.

Instead, keep it simple: run the containers as systemd units. Then, they can be upgraded easily using Ansible. The container images themselves can be copied manually then docker image import'ed. This has the added security benefit of no external dependencies.

[squeed]

Capturing a further discussion: If auto-updates (or simple updates) are desired, then a simpler design is possible:

1. Set up a trusted tuf / notary chain
2. New releases go to a "production-latest" image tag
3. A cron job on the server periodically refreshes the image and restarts if needed.

This means that the notary system is an extremely critical part of the security properties of the system - anyone who can push that tag can pwn every securedrop instance. Autoupdating is scary.

[heartsucker]

The current system uses cron-apt to pull the latest SecureDrop Debian packages which makes the single GPG signing key a critical part of the infra. Moving to Notary in its current incarnation (no support for thresholds) would be roughly equal in security. Also, I think the whole team would agree that auto updates are better than manual ones as we can get security critical patches out much, much faster. Given that, it's safe to say that autoupdates will stay. The mechanism and format of said upgrades is the current debate.

[squeed]

Another option is to distribute the SecureDrop unit file updates via cron-apt, and those reference the container image versions. While it does mean a more complicated update flow, it allows the docker parameters to be tied to the application version, in case things change.

[msheiny]

Changed the title of this ticket to reflect that CoreOS specifically will not be a good fit due to the difficulty of us providing our own kernel updates in the mix. However, we are still debating heavily bringing containers into the story, and updating them will still be a technical issue, so I think this ticket discussion is still relevant and am leaving it open.

[eloquence]

We've not made a final commitment to a container-based OS in general or CoreOS in particular. We'll re-evaluate our options after the transition to Ubuntu 20.04 is complete (#4768). See #5517. Closing this old issue and retiring the associated label for now.