Release SecureDrop Client 0.4.0

[eloquence]

Tentative release date: December 9, 2020

The next version of the SecureDrop Client will ship seen/unseen support at the source level, and will include small changes to the SecureDrop Client look and feel. This issue tracks the release-critical issues, the QA process and the release itself.

Starting with this release, we will be using a release/ branch model similar to SecureDrop Core.

• Land seen/unseen support Add Seen/Unseen support at source level #1165
• Land source list layout improvements Source list UI facelift #1174
• Final UX review w/ @ninavizz
• Prepare changelog
• Cut release branch
• Create test plan
• QA
• Production release

Description and test plan

The following PRs have landed since 0.3.0:

• Support read/unread and track who sees a file, message, or reply (Add Seen/Unseen support at source level #1165)
• No longer expire source test key (No longer expire test key #1180)
• Restyle source widgets and source selection indicators (Source widget ui polish #1191)
• No longer maximize client window if it's already open on login (only maximize if window not already displayed #1192)

Test Plan for #1165:

1. Basic read/unread
   • A source shows up as read (not bold) when the logged-in user clicks on the source
   • A source shows up as read (not bold) when the logged-in user uses the web JI to download all of the source conversation items
   • A source shows up as read when a new file or message is submitted by the source if the source is already selected in the client
   • A source shows up as unread when a new file or message is submitted by the source if the source is not already selected in the client
2. Global read/unread
   • A source shows up as read (not bold) when someone other than the logged-in user has read the source from another session of the client
   • A source shows up as read (not bold) when someone other than the logged-in user has read the source from the web JI
   • A source shows up as read (not bold) when someone has read a part of a conversation and another person has read the other part of the conversation.
3. Offline mode
   • When offline, all sources show up as read (not bold)

Test Plan for #1180

[N/A]

Test Plan for #1191

1. Log into the client and select a source
   • Source name turns blue for the selected source
2. Click another source, and another, and another
   • Only the source that is selected has a blue source name
3. Log out and verify the same behavior ^

Test Plan for #1192

1. Start the client in offline mode and change the main window size
   • The window is maximized
2. Sign in from the user auth menu
   • The window size does not change upon sign-in
3. Sign out form the user auth mentu
   • The window size does not change upon sign-in
4. Close the client and log in
   • The window is maximized

[eloquence]

1. Basic read/unread

   • A source shows up as read (not bold) when the logged-in user clicks on the source
   • A source shows up as read (not bold) when the logged-in user uses the web JI to download all of the source conversation items
   • A source shows up as read when a new file or message is submitted by the source if the source is already selected in the client
   • A source shows up as unread when a new file or message is submitted by the source if the source is not already selected in the client

2. Global read/unread

   • ❌ A source shows up as read (not bold) when someone other than the logged-in user has read the source from the web JI

This is where I ran into some issues, and @creviera and I spent a bit of time today trying to understand better what's going on. In a nutshell, I've observed two scenarios where the SecureDrop Client and the Journalist Interface show different "seen/unseen" information.

1. If I download submissions via a JI as a user who has never sent a reply, that user is not picked up during the sync, and their seen actions are never picked up by the SecureDrop Client. This relates to previous issues we've seen with users not being correctly updated (cf. Deleted user is not fully removed from client database after a sync #1143). Syncing via the /users endpoint instead of via reply attribution is still an open issue (Synchronize user data via /users endpoint #1157), and doing this should help resolve this entire category of problem.
   
   Unfortunately, this is not a terribly unlikely scenario -- we know that some users act primarily in "triage" responsibilities, without ever replying to sources. Those users' "seen" actions would not be picked up by other users via the SecureDrop Client, leading to inconsistencies.

2. Legacy submissions without seen records but with the downloaded Boolean set in the Journalist Interface are not correctly picked up as seen in the SecureDrop Client. The Journalist Interface does pick them up as seen. We've not tested the downloaded behavior extensively, and this impacts all historical submissions a news organization may have. The result is inconsistent information display between SecureDrop Client and JI.

Assuming other folks can repro, I think this likely pushes the release date forward a bit, though we may still be able to release before the holiday period.

[sssoleileraaa]

Amazing debugging @eloquence! Thank you for making it easy to understand and fix. It's kind of late so not 100% sure, but I think both issues are resolved in this PR: #1197 (comment). It's in a draft state because I need to write tests still.

This relates to previous issues we've seen with users not being correctly updated (cf. #1143). Syncing via the /users endpoint instead of via reply attribution is still an open issue (#1157), and doing this should help resolve this entire category of problem.

I still agree with this 100% even though we lucked out by by continuing to set the downloaded flag for submissions post SecureDrop 1.6.0, meaning, in this case, we don't need to know who the user is. In the future, I think we should clean this up a bit so that the downloaded flag is no longer used except for legacy data, but it turned out to be super useful since we haven't resolved #1157 yet.

[zenmonkeykstop]

test plan ** IN PROGRESS**

• env: staging w/ nightlies (v0.3.0-dev-20201209-060101)

Test Plan for #1165:

1. Basic read/unread
   • A source shows up as read (not bold) when the logged-in user clicks on the source
   • A source shows up as read (not bold) when the logged-in user uses the web JI to download all of the source conversation items
   • A source shows up as read when a new file or message is submitted by the source if the source is already selected in the client
   • A source shows up as unread when a new file or message is submitted by the source if the source is not already selected in the client
2. Global read/unread deferring until Client 0.4.0 qa bug fix #1197 lands
   • A source shows up as read (not bold) when someone other than the logged-in user has read the source from another session of the client
   • A source shows up as read (not bold) when someone other than the logged-in user has read the source from the web JI
   • A source shows up as read (not bold) when someone has read a part of a conversation and another person has read the other part of the conversation.
3. Offline mode
   • When offline, all sources show up as read (not bold)

Test Plan for #1180

[N/A]

Test Plan for #1191

1. Log into the client and select a source
   • Source name turns blue for the selected source
2. Click another source, and another, and another
   • Only the source that is selected has a blue source name
3. Log out and verify the same behavior ^
   • Verified

Test Plan for #1192

1. Start the client in offline mode and change the main window size
   • The window is maximized - unclear what this means - window starts maximized, can be resized after clicking the appropriate button in title bar
2. Sign in from the user auth menu
   • The window size does not change upon sign-in
3. Sign out form the user auth mentu
   • The window size does not change upon sign-out
4. Close the client and log in
   • The window is maximized

[eloquence]

Test Plan for #1165:

1. Basic read/unread
   • A source shows up as read (not bold) when the logged-in user clicks on the source
   • A source shows up as read (not bold) when the logged-in user uses the web JI to download all of the source conversation items
   • A source shows up as read when a new file or message is submitted by the source if the source is already selected in the client
   • A source shows up as unread when a new file or message is submitted by the source if the source is not already selected in the client
2. Global read/unread
   • A source shows up as read (not bold) when someone other than the logged-in user has read the source from another session of the client
   • A source shows up as read (not bold) when someone other than the logged-in user has read the source from the web JI
   • A source shows up as read (not bold) when someone has read a part of a conversation and another person has read the other part of the conversation.
3. Offline mode
   • When offline, all sources show up as read (not bold)

However, note that when going online, I have to wait until the next sync for the change to be correct again.

Test Plan for #1180

[N/A]

Test Plan for #1191

1. Log into the client and select a source
   • Source name turns blue for the selected source
2. Click another source, and another, and another
   • Only the source that is selected has a blue source name
3. Log out and verify the same behavior ^

Test Plan for #1192

1. Start the client in offline mode and change the main window size
   • The window is maximized
2. Sign in from the user auth menu
   • The window size does not change upon sign-in
3. Sign out form the user auth menu
   • The window size does not change upon sign-in
4. Close the client and log in
   • The window is maximized

(I'm assuming that by "Close the client and log in", you mean exiting the application and starting it again, then logging in again.)

[eloquence]

I've observed one potential bug while testing this step:

A source shows up as read (not bold) when someone other than the logged-in user has read the source from another session of the client

What I did:

1. Submit some stuff as a source
2. Wait for the client to sync, the new stuff is correctly displayed as unseen
3. Exit the client
4. On the same computer, start the client and log in as a different user
5. As the second user, select the new source to mark it as seen

What I observed is that the source changed state twice -- I think it was "seen -> unseen -> seen". Perhaps there was a race condition between sync and POST that caused the incorrect state to be displayed for a bit. Based on this observation, I would recommend a second tester focus on exploratory testing for race conditions w/ syncs. If the display self-corrects, it may be a minor issue that we can fix after release.

[sssoleileraaa]

I think it was "seen -> unseen -> seen"

I see this too when following your steps, but it doesn't require that you switch between users since I was able to repro logging back in as the same user. Also if you wait up to 15 seconds for the next sync to occur before logging out, you'll see that it doesn't happen.

The reason this is happening is because of the way we wait until the next sync to get back the seen data from the server before updating our local db. We could update the client db immediately, but this goes against our current practice. We've discussed updating the client db first and then syncing from client to server, but decided to hold off until the next phase of read/unread.

Here's why I don't think this is a blocking issue:

• Waiting until sync to update our local db is current practice
• It makes sense to create SeenFile, SeenMessage, and SeenReply entries when the seen request is successful, but we'll have to watch out for the MetadataSyncJob running in a separate thread, also potentially creating the same seen records if it gets back around the same time
• If we update the local db first, we've got to watch out for things like an undo operation, e.g. a "mark as unread" feature, which gets complicated because this is a global feature where anyone logged in can change the read state from the JI or another instance of the client connected to the same server. I think this is a larger change that we should discuss further, and also discuss whether or not we want to support saving changes locally while offline.
• it's an edge case that actually results in the correct state

[sssoleileraaa]

However, note that when going online, I have to wait until the next sync for the change to be correct again.

I don't think this is necessarily a blocking issue but it is fixed in this PR: #1199. Good catch.

[zenmonkeykstop]

test plan

• env: staging w/ nightlies (v0.3.0-dev-20201210-150736)

Test Plan for #1165:

1. Basic read/unread
   • A source shows up as read (not bold) when the logged-in user clicks on the source
   • A source shows up as read (not bold) when the logged-in user uses the web JI to download all of the source conversation items
   • A source shows up as read when a new file or message is submitted by the source if the source is already selected in the client
   • A source shows up as unread when a new file or message is submitted by the source if the source is not already selected in the client
2. Global read/unread (using a user that's never replied as the "someone other")
   • A source shows up as read (not bold) when someone other than the logged-in user has read the source from another session of the client
   • A source shows up as read (not bold) when someone other than the logged-in user has read the source from the web JI
   • A source shows up as read (not bold) when someone has read a part of a conversation and another person has read the other part of the conversation. not sure how to interpret this - since a source is flagged as read when the conversation pane is loaded, it seems like a duplicate case of the two above
3. Offline mode
   • When offline, all sources show up as read (not bold)

Test Plan for #1180

[N/A]

Test Plan for #1191

1. Log into the client and select a source
   • Source name turns blue for the selected source
2. Click another source, and another, and another
   • Only the source that is selected has a blue source name
3. Log out and verify the same behavior ^
   • Verified

Test Plan for #1192

1. Start the client in offline mode and change the main window size
   • The window is maximized
   • Window size is changeable via WM buttons/context menu
2. Sign in from the user auth menu
   • The window size does not change upon sign-in
3. Sign out form the user auth mentu
   • The window size does not change upon sign-out
4. Close the client and log in
   • The window is maximized

[eloquence]

(Completed last week.)