Proposal: Support Rust as a first‐class language

This was originally approved by the SecureDrop team on 2023-02-06

Proposal: Support Rust as a first-class language

Affected components

SecureDrop Server, SecureDrop Workstation, SecureDrop Client

People

Everyone who works on code

Problem Statement

Currently Python is our primary development language; it is a well known dynamic language that is usually straightforward to pick up. We also have a significant amount of expertise in Python and are able to work rather successfully with it. But it has some drawbacks:

1. Python is duck typed, which requires use of external tools (mypy, flake8, pylint) to try to bolt on strict typing
2. Dependency management is a mess, we have to go to extensive measures (managing our own wheels) to achieve pinned dependencies and reproducible builds.
3. Tooling is scattered, we typically need: black, isort, flake8, mypy, pylint, bandit, semgrep, pytest.
4. We are at the mercy of the Python version shipped by each distro we use. Because we work across multiple (Ubuntu, Debian, Qubes dom0/Fedora, Tails), it's an extra thing we need to keep in mind and adapt for.

On the other hand Rust has:

1. A very strong type system and libraries that utilize the type system to push errors to be compile-time rather than runtime.
2. Reasonable dependency management with cargo.
3. Most tooling is built-in, we just run cargo fmt, cargo clippy and cargo test.
4. Rust is a compiled language so we can pick the version of Rust we want and ship the binaries to whichever distro we need.

There are other benefits Rust has like absurdly better performance and memory safety, but I don't think those should be key motivations to switch, just extra icing on the cake. Largely we are constrained by I/O rather than execution speed and the Python code we write is mostly memory safe already (maybe not Qt??).

Drawbacks

Rust does have some drawbacks compared to Python. In no specific order:

• the Rust standard library is intentionally much smaller than Python's "batteries-included" approach. You could write a rather comprehensive CLI app in Python with no external dependencies, doing so in Rust would be pretty hard.
• some external libraries aren't necessarily as good, for example there's nothing anywhere close to SQLAlchemy+alembic in Rust. (This works both ways, there are some Rust projects that are wonderful and have no equivalent in Python!)
• dynamic things like monkeypatching (frequently used in tests) is straight-up impossible in Rust.

Solution impact

The biggest impact is that developers need to be familiar with Rust to be able to read and contribute to the code. Rust is generally harder to pick up because it uses a different paradigm of ownership and borrowing. But those up front costs are quickly paid off with increased developer happiness (we hope!).

For end users there should be no noticeable difference.

Requirements and constraints

1. We have figured out adequate QA checks for Rust code that runs locally and in CI
2. We are able to review/vet the trustworthiness of our dependencies
3. We are able to ship the code in reproducible debs and/or rpms.

Not discussed here as a requirement because it is a team consideration rather than a technical one, is that enough people need to be able to work on Rust code.

Guidelines

We should have guidelines on when it's appropriate to:

1. start a new project in Rust rather than Python (assuming our default language will continue to be Python)
2. port an entire project from Python to Rust
3. port parts of a project from Python to Rust

The Drawbacks section above lists some reasons why we'd prefer Python for some projects.

Exploration

Rust was first added to the securedrop package build toolchain for cryptography in June 2021 (https://github.com/freedomofpress/securedrop/issues/5898).

In April 2022 we started exploring the use of Sequoia-PGP (https://github.com/freedomofpress/securedrop/issues/6399) and out of that, started looking at how to review Rust dependencies (https://github.com/freedomofpress/securedrop/issues/6500).

Initial proposal

1. To start with, we will run rustfmt, clippy (with warnings causing failures), and cargo test. We'll also test against nightly Rust (in non-blocking mode) so we can identify potential problems with future Rust versions before they hit stable. And we can use LLVM tools to generate code coverage reports (see this blog post for an explanation on how).
2. We should be able to mostly adopt the diff review system, just with better tooling (cargo vet). We will also trust audits/reviews done by Firefox, since the integrity of Firefox (via Tor Browser) is already part of our trust model.
3. In theory cargo build --release should just work. Might need to adjust various flags or whatever. But we absolutely should not be shipping new non-reproducible code.

Guidelines

1. It's appropriate to start a new project in Rust rather than Python when:
   • The project is being shipped to users in a compiled package. For example, it's OK if it's being shipped as a deb, but would not be OK if it's being pulled via git on Tails and we try to compile it on the host.
   • Most or all Python dependencies have equivalents in Rust already. In other words, if a Rust version would have to implement non-trivial stuff that in Python would have been a dependency, that's not OK. (Within reason)
   • There is no significant impediment to other projects by using Rust. Given that most components talk to each other over HTTP or some RPC, it seems unlikely, but we don't want to start new project X in Rust and then realize it can't talk to existing project Y in Python. (See also the point below about simple vs complex types in hybrid Python+Rust projects)

Case study: ???

2. It's appropriate to port an entire project from Python to Rust when:
   • All of the above from part 1 apply.
   • There is some concrete benefit to using Rust, whether its specific bugs that Rust would've avoided, better performance, etc.
   • It is possible to finish the port in X amount of time (where X should be relative to the amount of benefit we get).

Case study: securedrop-proxy has come up a few times as something that could use a Rust rewrite. It is shipped to users as a deb, all dependencies exist in Rust already, and securedrop-client talks to it over RPC, so the implementation language should be transparent as far as the client is concerned. It's ~1.3k lines of Python, which should be reasonable to port to Rust in ~2 weeks.

As for concrete improvements, using Rust would've avoided https://github.com/freedomofpress/securedrop-security/issues/73 since we would've bubbled up the "missing keys" error (of course the same could've been done in Python...it just wasn't).

3. It's appropriate to port parts of a project from Python to Rust when:
   • All of the above from part 1 apply.
   • It is not possible or not desirable to port the entire project to Rust, but there is some concrete benefit in some parts being in Rust.
   • The component being ported to Rust is relatively isolated (or can be) from the rest of the project.
   • It's worth considering what kind of types need to be shared across the Python<-->Rust boundary. Simple types work best, while e.g. trying to interact with a SQLAlchemy model in Rust probably doesn't work very well.

Case study: SecureDrop server currently shells out to gpg via the pretty_bad_privacy library. Sequoia-PGP is a library-first re-implementation of PGP in Rust. It is shipped to users as a deb, the main dependency is Rust, and we'll use pyO3 to implement a the Rust<-->Python bridge.

It is infeasible to port all of SecureDrop to Rust, but there numerous deficiencies with pretty_bad_privacy that switching to Sequoia will be a significant improvement. The gpg-calling functions have already been mostly isolated via the EncryptionManager class. Everything will cross the Rust<-->Python bridge as String/str or Vec<u8>/bytes, no other types needed.

(Note that this idea is a separate proposal.)

Selected proposal

The above proposal described in "Initial proposal" was selected.