Packaging process builds Rust code

[legoktm]

Our make build-debs packaging process needs to build a Rust wheel and install it into the virtualenv. As part of this we need to make sure that the built wheel is fully reproducible so we aren't regressing on reproducible builds.

[legoktm]

Just noting that in my dev Qubes VM with 6 CPU cores assigned, it takes a little over 4 minutes to build redwood in release mode.

[legoktm]

So here's what maturin is doing under the hood, as discovered by RUST_LOG=debug maturin build -m redwood/Cargo.toml -v -r

2023-06-28T20:05:49.562717Z DEBUG maturin::project_layout: Using cargo manifest path from command line argument: "/home/user/github/freedomofpress/securedrop/redwood/Cargo.toml"
2023-06-28T20:05:49.562741Z DEBUG maturin::project_layout: Resolving cargo metadata from "/home/user/github/freedomofpress/securedrop/redwood/Cargo.toml"
2023-06-28T20:05:49.680016Z DEBUG maturin::project_layout: Found pyproject.toml at "/home/user/github/freedomofpress/securedrop/redwood/pyproject.toml"
2023-06-28T20:05:49.680486Z DEBUG maturin::project_layout: Resolving cargo metadata from "/home/user/github/freedomofpress/securedrop/redwood/Cargo.toml"
2023-06-28T20:05:49.798743Z DEBUG maturin::project_layout: Project layout resolved project_root=/home/user/github/freedomofpress/securedrop/redwood python_dir=/home/user/github/freedomofpress/securedrop/redwood rust_module=/home/user/github/freedomofpress/securedrop/redwood/redwood python_module=/home/user/github/freedomofpress/securedrop/redwood/redwood extension_name=redwood module_name=redwood
🔗 Found pyo3 bindings
2023-06-28T20:05:49.892508Z DEBUG maturin::python_interpreter: Found CPython interpreter at /home/user/github/freedomofpress/securedrop/.venv/bin/python3
🐍 Found CPython 3.8 at /home/user/github/freedomofpress/securedrop/.venv/bin/python3
📡 Using build options compatibility from pyproject.toml
2023-06-28T20:05:49.892782Z DEBUG maturin::compile: Setting PYO3_PYTHON to /home/user/github/freedomofpress/securedrop/.venv/bin/python3
2023-06-28T20:05:49.892800Z DEBUG maturin::compile: Running CARGO_ENCODED_RUSTFLAGS="-C\u{1f}link-arg=-fuse-ld=/usr/bin/mold" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.8-64bit" PYO3_PYTHON="/home/user/github/freedomofpress/securedrop/.venv/bin/python3" PYTHON_SYS_EXECUTABLE="/home/user/github/freedomofpress/securedrop/.venv/bin/python3" "cargo" "rustc" "--message-format" "json-render-diagnostics" "-v" "--manifest-path" "/home/user/github/freedomofpress/securedrop/redwood/Cargo.toml" "--release" "--lib"
...
2023-06-28T20:08:25.756120Z DEBUG maturin::module_writer: Adding redwood-0.1.0.dist-info/METADATA
2023-06-28T20:08:25.756369Z DEBUG maturin::module_writer: Adding redwood-0.1.0.dist-info/WHEEL
2023-06-28T20:08:25.756691Z DEBUG maturin::module_writer: Adding redwood/__init__.py
📖 Found type stub file at redwood.pyi
2023-06-28T20:08:25.756900Z DEBUG maturin::module_writer: Adding redwood/__init__.pyi from /home/user/github/freedomofpress/securedrop/redwood/redwood.pyi
2023-06-28T20:08:25.757153Z DEBUG maturin::module_writer: Adding redwood/py.typed
2023-06-28T20:08:25.757345Z DEBUG maturin::module_writer: Adding redwood/redwood.cpython-38-x86_64-linux-gnu.so from /home/user/.cargo/target/release/maturin/libredwood.so
2023-06-28T20:08:26.382540Z DEBUG maturin::module_writer: Adding redwood-0.1.0.dist-info/RECORD
📦 Built wheel for CPython 3.8 to /home/user/.cargo/target/wheels/redwood-0.1.0-cp38-cp38-linux_x86_64.whl

Here's what the __init__.py file contains:

from .redwood import *

__doc__ = redwood.__doc__
if hasattr(redwood, "__all__"):
    __all__ = redwood.__all__

for reference:

>>> redwood.__all__
['generate_source_key_pair', 'encrypt_message', 'encrypt_file', 'decrypt', 'RedwoodError']
>>> redwood.__doc__
'A Python module implemented in Rust.'

I thiiiiink we could just do this ourselves?? Will keep poking.

[legoktm]

It ended up being like 40 lines of Python, which is pretty good I think!

maturin does have some nice safety features, like it checks that the PyInit_redwood symbol is present (https://github.com/PyO3/maturin/blob/3ea1d0f165351d44f4c02afed98a226334401558/src/compile.rs#L539), so I think we should eventually switch back to it, but I think we can just wait until it lands in Debian - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=999850

And since it's statically compiled, hopefully we'll just be able to use it directly out of unstable/testing (might run into glibc version issues?).