Implementing a journalist client

[redshiftzero]

Currently the Qubes workstation has a much better user experience compared to the manual workflow. Eventually, we plan to deprecate the use of the Journalist Interface in Tor Browser in favor of a custom native application. @kushaldas, @joshuathayer and I discussed one way this client can be implemented, described briefly below.

There are two proposed applications to replicate the current workflow :

• securedrop-api-proxy - this application lives in the sd-journalist AppVM (which is connected to the internet via the sd-whonix VM). The securedrop-api-proxy is simply a qrexec service that is an API proxy: it passes requests and responses between the to the SecureDrop Journalist API server-side and a program running in the sd-svs VM. It does not need to have any UI.

• securedrop-client - this application lives in the sd-svs AppVM and is the application that has the user interface. The user interface would allow users to do all the same things they have previously done through the Journalist interface: read messages, send messages to sources, open submitted files from sources (see [issue incoming]), etc. The sd-svs AppVM has no internet connectivity and passes API requests/responses through securedrop-api-proxy in sd-journalist.

[eloquence]

A basic client will be part of the version to be audited. This initial client will still download submissions to the local filesystem, that is, you will still use a file manager to open a previously downloaded submission or message. However, the graphical client should handle all the key operations of the journalist UI:

• Login with 2FA
• View available submissions, including all metadata available in the web interface
• Reply to source
• Star submission
• Mark submission as read
• Download 1-n submissions/messages associated with a source
• Download collection of submissions/messages associated with a source
• Delete 1-n submissions/messages on server
• Delete collection of submissions/messages on server

All of these operations are already supported in the API available in the journalist-api branch of SecureDrop.

[conorsch]

Distilling a conversation with @joshuathayer about the practical realities of working on workstation code, we should:

• prioritize support for the container dev env in SecureDrop in order to host the API with a minimum of fuss;
• re-use the same Standalone VM (tentatively dubbed sd-dev) to store the git repositories for both SecureDrop proper and the SecureDrop Workstation during development;
• run the GUI client application code in development on sd-dev, for rapid feedback; and
• continue to support integration tests in the SecureDrop Workstation repository, so that code changes to the GUI client application can be built and deployed to the sd-journalist VM.

For now, we're recommending that developers use the SecureDrop "staging" VMs, provisioned on the API branch, in order to host the API, for use when developing the GUI client application. In so far as the API remains stable, that's acceptable, but in general, use of the containerized dev env would be useful to all folks running Qubes (covered by #106).

[joshuathayer]

On @conorsch 's third point above

run the GUI client application code in development on sd-dev, for rapid feedback

that's not naively supported in the architecture that @redshiftzero and @kushaldas and I came up with (and is diagrammed above). In our design, the client runs in a networkless VM, communicating with the server via a proxy running in a separate VM. The proxy accepts qubes-rpc calls from the client and makes corresponding HTTP requests to the server.

So, simply running the client in the same VM as the server container isn't sufficient: we'd also need to install and configure the proxy in that same VM. When the client wants to make a request against the server, it would make a qubes-rpc call to its configured proxy, which happens to end up on the same VM. The proxy would make an HTTP call to its configured server, which happens to be in a container on the same VM. The server's response flows back from the container to the proxy, then back to the client.

In this way we're able to develop the server, client, and proxy all on the same VM. We could follow the same pattern if we want for sd-decrypt: it also can be developed on sd-dev if its qrexec policies are set correctly.

Of course we'd still develop salt configurations to create each component's VM, deploy the code to the correct places, and configure qrexec policies for each, but that becomes a somewhat distinct project from the "coding" part of the project where quick iteration is key.

[redshiftzero]

Dropping some initial thoughts about the implementation of the securedrop-client. I'm trying to distinguish here between things we need for initial prototype / security audit later this year and what we would eventually want in an ideal view (see #89 and #102).

Two basic points:

1. We should have a local database for the prototype for the audit. We should have (at least) two components: a storage layer, which uses SQLAlchemy (since we have good knowledge of it on the team), and a UI layer, which uses Qt. Storage layer sends/receives API requests/responses to/from VM containing the SD api proxy qrexec service.
2. We should not get ourselves into a situation where state is stored locally on the workstations that cannot be refetched (or relatively quickly recomputed from the downloaded content) from the server. For example, if we implement a notes feature, we must allow for the notes to be stored encrypted on the server, such that a journalist that e.g. gets a new device, can quickly resync state from the server.

In the thoughts that follow, “initially” means it’s something we should have for the initial prototype, “eventually” is something that will come in a future version

When client program starts up:

1. User logs in.
2. API query is submitted to /sources endpoint. We update the local database using the data provided in the response: new sources, new submissions are added to the local db.
3. New messages are immediately fetched and downloaded and decrypted for usability reasons. Otherwise documents are only downloaded on demand as they can be very large.
4. When journalist wants to read a document, they click download. After download, they click the open button, which will subprocess out to decrypt and open that document in the dispVM.
5. Eventually: We may want some kind of helper UI in the dispVM (TBD)

Local database should contain:

• Source table:

  • Merely store everything provided by API

• Reply table:

  • Initially: show which journalist sent a message to which source, kept encrypted
  • Eventually: such that journalists can more easily reconstruct conversations:
    • Replies should be encrypted both to the submission key and the source key
    • Replies should be marked as “read” by sources but not deleted (i.e. when a source deletes a reply it is really marking it as read). The timestamps on all replies for a given source should be set to the latest reply as is done for source submissions.

• Submission table:

  • Initially: merely save everything provided by API
  • Eventually: we store some metadata in the database about the submission for display in the UI (file type, possibly length of file e.g. 200 pages, 4:20 minutes, etc., possibly file name, etc)

• Eventually other data can be added like:

  • Notes
  • Messages to other journalists
  • Per-journalist metadata e.g. “favorite sources” (stars are for the entire application), whether a document has been opened (this replies the interface-wide read status)
  • Important: all of this content will need to be encrypted and synced to the SecureDrop server so that a user who destroys their workstation can recover.

Open questions:

• Should we immediately decrypt messages and store decrypted in sd-svs VM? (improves usability a lot as a journalist can easily read the conversation)

[redshiftzero]

here's a more detailed structure for the initial client (I'm going to make a separate repo for the client and start breaking this up so we can focus the discussion down a bit on the next few steps):

• Integrate alembic from the start to enable easier changes down the road
• Continue to use SQLite for simplicity (I know that SQLite has its challenges, but a easy to install/deploy client application like what we’re creating here is a reasonable use of SQLite imho)
• Where possible, clients follow URIs provided by API to reduce client/server coupling
• All API requests responses proxied through qrexec API proxy (Create an API proxy as a qrexec service #107)
• Blocking actions e.g. sync, will be executed in a separate QThread
• Archiving/cold storage of sources is done at a per-source level, not per submission (if user feedback indicates that the latter is useful, then we can implement it later)

Storage/controller layer

This layer will basically control the models which describes the database (below) so that functionally the Qt parts of our code are interacting with storage, not sqlalchemy directly, and act a presentation/view layer

Utility functions

sync

Sends request to /api/v1/sources/ endpoint and /api/v1/submissions/ endpoint

For each source uuid:

• If uuid does not exist in local db, create it and save all attributes
• else, if uuid does exist in local db but not server:
  • mark as pending deletion locally if source is not archived
  • else keep around if source is archived
  • if row still exists, compare attributes, update local database row with any changes

Note that deleting a source locally should delete the submissions on disk (i.e. cascade delete).

For each submission uuid:

• If uuid does not exist in local db, create it and save all attributes (but do NOT download)
• else, if uuid does exist in local db but not the server:
  • mark as pending deletion locally if corresponding source is not archived
  • else keep around if corresponding source is marked as archived
  • if row still exists, compare attributes, keep local database row updated with any changes

send

Subprocesses out to qrexec service to send a request. Clients wait until they get confirmation from the server that the request was successful prior to committing changes in the local database.

filter_by(journalist_designation)

This is for the search box that enables a user to find a specific source. It should:

• Sanitizes text provided by journalist in filter field
• Filters sources by this text, returns a list of sources for display in the UI

User related information

Anything related to login - token, expiration - all gets stored in memory (related: we should investigate swap situation on Qubes workstation)

Source

Class with the following methods:

• add
• delete
• archive - mark as cold storage
• unarchive - unmark as cold storage (if the journalist is done with the source)
• star
• unstar
• pending_deletion -> this should create UI notification for the user to confirm they want it to be deleted locally since it has been deleted on the server

Submission

Class with the following methods:

• delete
• download
• open -> subprocesses out to begin Qubes submission handling process

Reply

Class with the following methods:

• add - note: until Enable journalists to view, download and decrypt replies deleted from source interface securedrop#3097 is addressed (see notes on required API endpoints and other server side changes in that ticket), this will be add only
• open -> even initially we should save an encrypted copy on the local workstation (encrypted to the submission key), this open method should subprocess out to begin Qubes submission handling process

Database/models layer

sources table

• uuid
• journalist_designation
• is_flagged
• last_updated
• public_key
• interaction_count
• Is_starred (not using separate table here as we do on the server, there is no reason to)

submissions table

• uuid
• is_read (the number of submissions that have this designation is basically going to generate the initial “read” icon locally)
• filename -> path on the SVS where the doc is stored
• is_downloaded (we could also expose this in the sources list in the UI)
• size (expose in UI to give the user an idea of how long it will take to download)
• source_id -> id in local database

replies table

• source_id -> id in local database
• journalist_id -> id in local database
• filename

journalists table

• username -> to begin with, this will just be the current user (when replies are surfaced in the API via Enable journalists to view, download and decrypt replies deleted from source interface securedrop#3097 then we should store the usernames so it's clear who is talking to who)

[redshiftzero]

this was implemented, closing:
api proxy is in https://github.com/freedomofpress/securedrop-proxy
GUI is in https://github.com/freedomofpress/securedrop-client