Use Sequoia-PGP instead of gpg/pretty_bad_protocol

[legoktm]

Status

A branch, oxidize, contains the current state of progress.

Original description

We should consider using https://sequoia-pgp.org/ for SecureDrop's encryption instead of our current setup of the unmaintained pretty_bad_privacy library shelling out to gpg.

Recent issues with gpg like #6389 indicate that we really don't need most of gpg's functionality and our reliance on an unmaintained library that the author discourages people from using (as implied in the name). Using Sequoia as a library should allow us to use only what we actually need to use. We would also no longer be tied to the gpg on-disk keyring, so we could put them in the database, avoiding issues like #6270 (we'd need some migration process of course).

According to https://sequoia-pgp.org/status/, no external audit of the code has been done yet.

We will also want to look into whether we can reuse something like johnnycanencrypt or whether we should just roll our own pyO3 bindings.

Subtasks

• Determine migration plan for gpg -> Sequoia #6499
• Write code (including tests for all of the below):
  • New source creation uses Sequoia #6799
  • One-time migration of sources' public keys and fingerprints from gpg to database-backed storage #6800
  • On-demand migration of sources' private keys to database-backed storage upon login #6802
  • Vendor pretty-bad-protocol and strip it down to what is needed #6807
  • BLOCKER: cumulative dependency/diff review, including Determine plan for auditing Rust dependencies #6500
• request review by Sequoia team #6801
• formally approve Sequoia as a critical dependency #6806
• Rust integration
  • CI runs Rust-specific lint/test jobs #6814
  • Development environment builds Rust code (optionally with live-reload) #6816
  • Packaging process builds Rust code #6817
  • Dependency alerts are configured using RustSec/cargo audit #6815

[legoktm]

My current thinking is that we use pyO3 to write a very slim bridge to sequoia that just implements the bare functionality of what we need.

We might want to maintain this bridge in a separate repository, build it as a wheel and treat it as a dependency just so we don't need to add the Rust toolchain and build step into the dev container, make build-debs, etc. It would make integration testing annoying though since you have to keep manually building and installing the wheel.

Or we just bite the bullet and integrate all the Rust stuff in properly, which, as a side-benefit lowers the bar for the next person trying to introduce Rust things :)

[legoktm]

We would also no longer be tied to the gpg on-disk keyring, so we could put them in the database

There is a minor downside to this, by putting private keys in the database it would just take an SQLi to access them, rather than an arbitrary file read or RCE. That said I think our risk of SQLi is very low because of how we use SQLAlchemy for just about everything, I just wanted to mention that it's a slight risk.

[legoktm]

by putting private keys in the database it would just take an SQLi to access them

If we're seriously worried about this (I'm not), we could encrypt the keys before putting them in the database with a fixed secret.

---

I pushed some WIP of the Rust code to the "oxidize" branch: https://github.com/freedomofpress/securedrop/tree/oxidize. Key generation and encryption was very straightforward to implement...decryption looks very complicated so that's for tomorrow.

[legoktm]

One weird thing with using pyO3 is how bytes are handled at the Rust/Python boundary. If you have a Rust String, it turns into a Python str and vice-versa. Now, a Python bytes gets turned into a Rust Vec<u8>, which if you return, turns into a Python List[int], oops. There's a PyBytes type you can use to return real Python bytes, but it requires that the function holds the GIL So any function that returns bytes (currently key generation and decryption) will hold the GIL, and the other ones won't (I think).

In any case this is all academic, because even if we're not holding the GIL I don't think it'll have any practical impact on our program since it's all single-threaded I believe. But mostly writing this out because it took me a while to figure out how to return real Python bytes.

[legoktm]

I mostly have all of securedrop wired to use redwood in the dev container, some parts work and then something else has encoding messed up and is displaying gibberish ascii symbols instead of the decrypted text. I'm tempted to pass everything as String/str over the bridge since that works fine and we know that it's all unicode since we convert it on the Python side unconditionally!

The big TODO that's left is sticking the journalist public key somewhere that isn't the GPG keyring. Probably just a file on disk I think.

---

The integration with the dev container mostly works fine, though I don't have live-reload set up yet. cargo build (hidden by maturin build) has to download the entire crates.io registry each time the container starts which takes like a minute or two even if the build itself takes a fraction of a second. I think we can have it use some directory on the host as the cache and mount it into the container so it persists a bit more.

[legoktm]

I switched to using String/str everywhere and now encryption/decryption seems to all work fine! (encrypting files doesn't work yet, will get to that tomorrow hopefully)

[legoktm]

Some bad news on the migration front. We can't export secret keys out of GPG without knowing the passphrase. This is because GPG stores them using a different internal format (https://unix.stackexchange.com/questions/698117/export-gpg-private-key-without-knowning-the-passphrase) so it has to decrypt them before it can export them in OpenPGP format.

This means we can't do a full offline migration, it can only be done when sources log in and we have their passphrase. So we have two options, 1) do the migration whenever a source logs in, or 2) just have "old gpg sources" and "new sequoia sources" that just vary on which code path is hit.

My current thinking is that we migrate the public key offline, and then when the source logs in we migrate the secret key (option 1). This minimizes the use of gpg overall, since post-login we shift everything over to sequoia. Also if a journalist replies before the source logs in, it still uses sequoia since we've migrated the public key.

[legoktm]

High-level roadmap for remaining todos:

• Figure out plan for migration
• Write the code for the migration
• Ask Sequoia team for code review, also solicit any questions from SD team members about overall health of project, known risks, etc.
• Get tests passing, add new tests as needed
• Integrate into packaging process
• Set up security scanning using cargo audit
• Figure out what our code review plan for Rust deps is - maybe using Mozilla's cargo-vet? https://lwn.net/Articles/897435/

[zenmonkeykstop]

The downside of both options is that we can't really define a cutoff for removal of the old gpg code - it will be hard to verify that all valid/active sources have been migrated. I wonder if it's possible to import the encrypted key without decrypting it, maybe skipping python_gnupg and parsing the keyring directly with https://docs.sequoia-pgp.org/sequoia_openpgp/cert/prelude/struct.CertParser.html or similar.

(Stuff like #6488 to clean up dbs and keyrings would probably help reduce the pain a little.)

[legoktm]

I've split discussion of the migration process to #6499.

And I turned my remaining todos into a checklist in the task description.

[rocodes]

Apologies for being so late to this conversation, I'm still catching up on the roadmap and reading here, but I would be interested in nominating the migration plan as something that maybe we can get formal review from one or (more of) @lsd-cat or @L3th3 before we proceed to the migration phase (I think it's covered in task item 3, but I would kind of argue for a first pass of the security review before too much code is written).

[nwalfield]

Some bad news on the migration front. We can't export secret keys out of GPG without knowing the passphrase. This is because GPG stores them using a different internal format (https://unix.stackexchange.com/questions/698117/export-gpg-private-key-without-knowning-the-passphrase) so it has to decrypt them before it can export them in OpenPGP format.

This means we can't do a full offline migration, it can only be done when sources log in and we have their passphrase. So we have two options, 1) do the migration whenever a source logs in, or 2) just have "old gpg sources" and "new sequoia sources" that just vary on which code path is hit.

My current thinking is that we migrate the public key offline, and then when the source logs in we migrate the secret key (option 1). This minimizes the use of gpg overall, since post-login we shift everything over to sequoia. Also if a journalist replies before the source logs in, it still uses sequoia since we've migrated the public key.

I opened an issue about this: https://gitlab.com/sequoia-pgp/sequoia/-/issues/928

[legoktm]

With the release of 2.7.0, this is now done :)