Qubes Journalist Workstation Testing: 2nd Round (Q4 2018)
- 2/3 of participants were estimated to be between 36 and 75 years old.
- 1/4 of our participants were women, 1 European PoC, and 1 American PoC.
- 17% of our participants work outside North America.
- 5 folks that have never used SecureDrop, 12 existing users.
- 4 freelance journalists, and 13 newsroom staffers from 7 newsrooms
- Among the 7 newsrooms using SecureDrop, each has trained an average of 4.3 journalists, but only an average of 1.5 are regularly "willing to use SecureDrop."
- 4 of the 7 newsrooms have journalists across multiple offices using or needing SecureDrop resources.
- Every-other-week appears to be the average rate at which participants check their current SDs.
- Two newsrooms check very infrequently, and the rest between every few days and every day.
- More About Testing Participants in more detail (see Participants tabs at bottom of GSheet).
-
Spreadsheet tracking flows, features, and functionality being looked at.
- Detailed findings and insights are in the "Testing Insights Matrix" tab.
- Fancy charts up above are from more detailed info on test participants to date, on the second tab of the spreadsheet
- 2 participants (including last week's)
First participant
—will do—Second participant findings
—tbd—
- 1 Participant
NO FINDINGS: Prototype(s) buggy, participant offered to reschedule
- Participant had established repor w/ FPF folks via past trainings
- Prototype files had been fully refactored to address issues from prior week; despite that, one minor bug proved to be a roadblock.
- Tools capability to mimic Qubes dropdown/flyout in connecting a USB drive, proving very difficult to accurately reproduce in Nina's WYSIWYG prototyping tools (yep, plural).
- Nina/Erik delayed further scheduling for next week, to prioritize fully smoothing-out prototypes this week.
- Staff editor
- 10-15yrs as a journo
- Tech fluence: Expert: I'm comfortable writing code or taking things apart to see how they work
- Two-Factor Authentication, PGP/GPG encryption, Full-disk encryption, GitHub, Signal, Tor, Windows, mac OS Familiar with SD
- Discussion Guide w/ Prototype Links
Top Findings
.3rd Participant
- I was surprised at how little encryption seemed to be happening"
- "Interface felt sleek and tidy"
- Need to resize to fit prototype better in browser window (waaay shrink!)
- Overall strong concidence in system
- Positive iteration on attach wizard; an improvement. Possible TMI for a single screen?
- For later refinement of app: Messaging in "Save As" should be smarter, more caveating around this stuff; to revisit once braoder app more resolved.
2nd & 1st Participants
- Instructions— could refine wrt #2
- BACKUP PLAN!! Must always have one, be more careful to not over-write older versions of prototypes.
- Font sizing needs to be more easily legible for users for prototype, while full frame fitting into window better; bigger to-do
- Export flow remains troublesome; second user (ESL) REALLY struggled
- Paper airplane button (knockout on the SD qube) "is obtuse"
- How much are we evoking Email vs Chat?
- 4 Participants
- None, existing SecureDrop Users
-
First user
- From South Africa but currently in NA
- Freelancer
- Not familiar with VMs.
- Uses WhatsApp for encrypted comms.
- Only just heard about SD through survey.
-
Second user
- In South America
- Linux user
- Familiar with VMs, Encryption, etc.; Whonix
- Not highly technical
- Uses Mailvelope for encrypted comms.
- Familiar with SD and had previously spoken with Micah about possibly getting it for his newsroom.
-
Third user
- In North America
- Linux user
- Familiar with VMs, Mac user
- Moderately technical
- Reporter for a local corp-owned newsroom; not much IJ experience, it seemed (didn't ask).
- Prototype performance issues with this one.
- Findings roughed-out in "Testing Insights Matrix" tab of spreadsheet, yet to cut/paste quotes. Noteworthy moments are summarized in awful all-caps, though.
- Discussion Guide w/ Prototype Links (same as prior session; new prototype is still too buggy to test from).
Top Findings
-
- Export wizard remains stumbly
- Unchanged message at top, odd
- Absent functionality (in prototype) to execute pairing, confusing
- Both know about 2FA (via banking apps)
- Both interpreted "Safe USB" drive well
- Both were unfamiliar with encryption, but guessed what it was well enough
- Slight confusion with mixed messaging/email paradigm
- First user was entirely unable to find "Compose Reply to Benign Artichoke" block
- Graphic of paper airplane against 2-hue Qube, identified as 'unclear'
- No immediate action on this, tbd
- Good to include mention of "non-profit/foss" in intro-blurb to non-users
- Should develop a contingency plan for users with sloooooowwwwwww connections
- 2 Participants
- Both "Somewhat Comfortable," older users
- Neither, existing SecureDrop Users
- Discussion Guide w/ Prototype Links
- Planning
- Research plan
- Discussion guide & nav path outline
- Invision prototype
- Online location (Erik's GMeet—links vary with participant invitations)
Findings & Action Items
-
Hi/lo observations from testing, below. Detailed findings are in the linked GDoc; fully anonymized, and open for comments.
- Not clear. Most significant actionable issue between both participants.
- Subtlety in colors, italicization, etc., totally washed-over both users' heads
- Combined with confusion around encryption/decryption status of shown content, both emerged in parallel as highest priorities for next iteration.
- Seeing unread messages in Offline mode, confusing—as workflows unlikely to reproduce in reality.
- Next Iteration Action(s):
- 2-state Offline/Online banner w/ "Go Online" and "Last Refresh" and a hard-refresh button, include
- Interstitial sign-in pane between clicking the app icon and initial client view; Erik's ASCII sketch for reference
- Remove "Unread" messages from initial view in Offline mode
- Need messaging to confirm successful action completion, and where actions took place (local, server, both)
- "Conversations" concept confusing (Note: bucket nested within Sources list, to separate newsroom-reciprocated correspondence from un-reciprocated submissions)
- Timestamp(s) in messaging pane too small/pale
- Discoverability issues with "Reply To" block
- Pointers on message bubbles, confusing
- Language: "Export" more functionally clear than "Save"
- Attached file things generally clear, some bumps to smooth-out
- Idea validated that "Pick things to delete" as a dropdown item on a single Source, would be useful.
- Next Iteration Action(s):
- Include toast (transient notification) messaging—initially, for "Delete" flow, and with explicit messaging
- Update "Delete Source" language to "Delete Account"
- Remove "Conversations" bucket (currently nested within Source's list)
- Simplify read/unread paradigm, with removed "offline" state and "Conversations"
- Change all uses of "Save" to "Export"
- Add Download icon, and omit word "Decrypt" in initial view—but note decryption happening in-progress.
- Include "Latest Action" verbiage on messaging pane timestamp, and improve visible discovery
- Include big paper-airplane-like button in "Reply To" field
- Kill the bubble pointers, make less bubbly-looking
- Research plan
- Discussion guide & Nav Path Outline
- Prototype
- Online location (Erik's GMeet—links vary with participant invitations)
- Existing customers, Erik handling
- Correspondence messaging
-
Kickoff
- Meet to revisit goals, methods, and timing
- Shift goals for this iteration?
- Things to do differently?
- Different users to recruit?
- Meet to revisit goals, methods, and timing
-
Preparation Communication Train
- New Moderator/Note-taker pairs? Anyone interested in observation?
- Schedule session blocks
- Outreach to prospective Participants
- Add new participants to spreadsheets
- GCal invites to Participants, Moderator(s), Note-taker(s), and optional Observer(s)
- Follow-up Moderator introduction email
-
Session Preparation
- Updates to Research Plan?
- New Session Outline
- Update Discussion Guide
- New Materials Assembly (prototype, cards, etc)
- Internal meeting to communicate prototype updates & new interest points w/ pairs
- Per-user, customized/password-protected prototypes generated?
- Conduct sessions
- Post-session synthesis
-
Wrapping-Up
- Complete findings/synthesis synopsis; post to wiki
- Compose dev/design action-items; share-out where appropriate (as GitHub Issue or in UxD wiki)
- Post Template Prototype to wiki
- Post Discussion Guide, Research Plan, Session Outline to wiki
- Everything above: final artifacts attached here as links?
- Write/Send thank you notes to Participants
- Get rid of this damn checklist
- Present/debrief findings to team
- Post findings to Gitter and Forums
-
Kickoff
- Create wiki page
- Meet to discuss goals, methods, and timing
- One-off study, or a series?
- Timing of synthesis/findings/iteration cycles
- How many sessions per iteration: where will the value be?
-
Preparation Communication Train
- Create Study Spreadsheet to track all participants
- Internal outreach to establish Moderator/Note-taker pairs, and to solicit interest in optional observers
- Schedule session blocks
- Initial outreach to prospective Participants
- GCal invites to Participants, Moderator(s), Note-taker(s), and optional Observer(s)
- Follow-up Moderator introduction email
-
Session Preparation
- Create wiki page
- Begin Research Plan
- Begin Session Outline
- Begin Discussion Guide
- Begin Materials Assembly (prototype, cards, etc)
- Complete Plan & Outline
- Internal session dry-run
- Complete Discussion Guide + Template Prototype
- Per-user, customized/password-protected prototypes generated?
- Conduct sessions
- Post-session synthesis
-
Wrapping-Up
- Complete findings/synthesis synopsis; post to wiki
- Compose dev/design action-items; share-out where appropriate (as GitHub Issue or in UxD wiki)
- Post Template Prototype to wiki
- Post Discussion Guide, Research Plan, Session Outline to wiki
- Everything above: final artifacts attached here as links?
- Write/Send thank you notes to Participants
- Get rid of this damn checklist
- Present/debrief findings to team
- Post findings to Gitter and Forums