Home

Work in progress

This page serves two purposes:

1. Describe abstractly what our interactive (CLI) and automated (CI) tools are for across the repositories that compose the SecureDrop project.
   • This is in contrast to any given repository's readme, which describes concretely how to use these tools in that context.
2. Track (a) inconsistencies to resolve across repositories and (b) new features, goals, or principles to implement.

While some of this material may eventually be reflected in the developer documentation, for now it is a (public) "living document" rather than a releasable documentation artifact.

Goals and principles

• It should be easy to run the same thing that CI is running. Conversely, CI should run what you're running locally.
• Similarly-named jobs and targets (e.g., lint and make lint) should do the same thing, following the principle of least surprise.
• Standard tooling (e.g. ShellCheck) should be available opportunistically: even if a project has no shell scripts yet, ShellCheck should kick in automatically if it acquires one.
• We're using Make as a command-runner, not for any real build logic.
  • Make's main pro is that it's universal, everyone knows "make test", etc. and already has it installed.
  • We do use make as a build tool for client localization targets, but that's not necessary.
• We should use tool-specific configuration (e.g. Black, pytest) rather than committing complex command invocations to Make targets.
  • Makes it easier for new developers who already have experience using the tool to also use the tool here.
  • Helps with other tooling, e.g. IDEs that know how to use the tool in the standard way (e.g. PyCharm can run black to autoformat every time you save a file), but won't know about our wrapper that contains our configuration arguments.
• Non-trivial changes should (1) originate and get tested in project repositories; (2) get proposed formally in a pull request in securedrop-tooling; and finally (3) propagate back out to project repositories. The DX working group can help you do this!

CI overview

Across projects/repositories, we use CI to accomplish the following tasks in the following ways (including the following exceptions/gaps):

Repository	securedrop	securedrop-client	securedrop-proxy	securedrop-export	securedrop-log	securedrop-workstation	securedrop-updater
CI image		debian	circleci/python:3.7-buster	debian	debian		fedora
Python version
Checks ShellCheck	yes						yes
...on any and all shell files present, or passes if none
Checks mypy	yes, config partially in shell wrapper	yes, config in Makefile	yes, config in Makefile	no	no		no
...in strict mode, including Qt
Runs tests via pytest	yes	yes, with xvfb-run	no, unittest	yes	no, unittest	no, unittest	yes
Checks Black+isort	yes, config in Makefile	yes	yes, config in Makefile	yes black w/ config in Makefile, no isort	no		yes
Runs flake8	yes	yes	yes	yes	no		yes
Checks safety	yes, complex Makefile command	yes, complex Makefile command	yes, complex Makefile command	yes, complex Makefile command	yes, complex Makefile command		no (but no prod deps)
Extracts source strings for localization		yes	no	no	no		no
...if babel.cfg is present, otherwise passes
Caches CI jobs	yes, but inefficiently
CI jobs are grouped in workflows
CI jobs are parameterized in matrices
...

Tooling overview

Across projects/repositories, we use Makefiles and shell scripts to accomplish the following tasks in the following ways (including the following exceptions/gaps):

Repository	securedrop	securedrop-client	securedrop-proxy	securedrop-export	securedrop-log	securedrop-workstation	securedrop-updater[^1]
Self-documenting	make help	make help	make help	make help	make help	make help	make help
Report version		make version
Check environment						make validate
Install non-Python dependencies						make install-deps	make install-deps
Run in a development environment	make dev[-tor], via script in container	./run.sh				make dev, via scripts
Load development data	loaddata.py	create_dev_data.py
Run in a staging environment	make staging, via script/Molecule					make staging, via scripts
Management	manage.py
Create a Python venv	make venv	make venv{,-mac,-sdw}	make venv	make venv	make venv	make venv	make venv
Set up Git hooks	make hooks (via make venv)	make hooks (via make venv)
Clean up	make clean, via script	make clean	make clean			make clean
Update Python requirements	make update-pip-requirements	make requirements	make update-dependency → make requirements	make update-dependency → make requirements	make update-dependency → make requirements	make update-pip-requirements	make update-pip-requirements
Check ShellCheck	make shellcheck, via script						make shellcheck, via script
Check mypy	make typelint, via script	make mypy, config in Makefile	make mypy, config in Makefile
Run tests	make test, via script in container	make test{,-functional,-integration,-random}, via xvfb-run	make test, via coverage	make test	pytest	make test, via unittest	make test, in container
Check Black + isort	make check-{black,isort}, config in Makefile	make check-{black,isort}, config in Makefile	make check-{black,isort}, config in Makefile	make check-{black,isort}, config in Makefile			make check-{black,isort}, config in Makefile
Apply Black + isort	make {black,isort}	make {black,isort}	make {black,isort}	make black (no isort)		make black (no isort)	make {black,isort}
Check flake8	make flake8	make lint	make lint	make lint		make flake8	make flake8
Check safety	make safety, complex Makefile command	make safety, complex Makefile command	make safety, complex Makefile command	make safety, complex Makefile command	make safety, complex Makefile command
Check Semgrep	make semgrep, complex Makefile command	make semgrep, with arguments from Makefile	make semgrep, with arguments from Makefile
Additional linting	Ansible, Bandit, pylint; HTML, YAML	Bandit	Bandit		Bandit		Bandit, rpmlint
Check source strings are up to date		make check-strings
Extract source strings for localization	make translate, via script in container	make extract-strings
Generate browsable documentation		make docs
Package	make build-debs, via script	via securedrop-builder	via securedrop-builder	via securedrop-builder	via securedrop-builder	via securedrop-builder	make build-rpm, via script

[^1]: As of 31 January 2023.

requirements.txt management

Makefile targets, wrapper scripts and layout of requirements.txt files...

safety

Here's what we have in securedrop-client:

IGNORED_VULNERABILITIES ?= "51668"

.PHONY: safety
safety: ## Runs `safety check` to check python dependencies for vulnerabilities
	pip install --upgrade safety && \
		for req_file in `find . -type f -wholename '*requirements.txt'`; do \
			echo "Checking file $$req_file" \
			&& safety check --full-report --ignore $(IGNORED_VULNERABILITIES) -r $$req_file \
			&& echo -e '\n' \
			|| exit 1; \
		done

• only client uses "IGNORED_VULNERABILITIES", the other components don't
• A simpler way to write this is cat requirements/*requirements.txt | safety check --full-report.
• Also unclear if safety should be pinned or not.
• Why do we not use the policy file thing? https://docs.pyup.io/docs/safety-20-policy-file What about the new project.json?