Skip to content

Qubes Journalist Workstation Testing: 2nd Round (Q4 2018)

Jump to bottom
Nina Eleanor Alter edited this page Nov 19, 2018 · 37 revisions

Eleven users, to date. They break-down, accordingly:

  • 2 folks that have never used SecureDrop, 9 existing users.
  • Majority so far have been mid-aged, mid/late career newsroom vets. Even the IT and support folks.
  • Three major newsrooms, two non-profits, one imprint(?) newsroom, 2 freelancers.
  • 1/3 of our participants were women, 1 European PoC, and 1 American PoC.
  • Every-other-week appears to be the average rate at which participants check their current SDs.
    • Two newsrooms check very infrequently, and the rest between every few days and every day.
  • To date we have not tested with any "Expert" or "Not Comfortable" users; those values are null, but shown in the Tech Savvy chart as representative values. Values were also assigned to users post-mortem, not self-assigned.
  • More About Testing Participants in more detail (see Participants tabs at bottom of GSheet).

What's being tested, and how'd it do?

  • Spreadsheet tracking flows, features, and functionality being looked at.
    • Detailed findings and insights are in the "Testing Insights Matrix" tab.
    • Fancy charts up above are from more detailed info on test participants to date, on the second tab of the spreadsheet

Iterative Studies

2.3 Study: Week of 19 Nov

Top Findings .
  • 2 Participants
  • Neither, existing SecureDrop Users
  • All findings, TBD

2.2 Study: Week of 30 Oct

Top Findings

-

  • Export wizard remains stumbly
    • Unchanged message at top, odd
    • Absent functionality (in prototype) to execute pairing, confusing
  • Both know about 2FA (via banking apps)
  • Both interpreted "Safe USB" drive well
  • Both were unfamiliar with encryption, but guessed what it was well enough
  • Slight confusion with mixed messaging/email paradigm
    • First user was entirely unable to find "Compose Reply to Benign Artichoke" block
    • Graphic of paper airplane against 2-hue Qube, identified as 'unclear'
    • No immediate action on this, tbd

Afterword:

  • Good to include mention of "non-profit/foss" in intro-blurb to non-users
  • Should develop a contingency plan for users with sloooooowwwwwww connections

2.1 Study: Week of 08 Oct

2.0 Study: Week of 01 Oct

Findings & Action Items

-

Hi/lo observations from testing, below. Detailed findings are in the linked GDoc; fully anonymized, and open for comments.

Online/Offline

  • Not clear. Most significant actionable issue between both participants.
  • Subtlety in colors, italicization, etc., totally washed-over both users' heads
  • Combined with confusion around encryption/decryption status of shown content, both emerged in parallel as highest priorities for next iteration.
  • Seeing unread messages in Offline mode, confusing—as workflows unlikely to reproduce in reality.
  • Next Iteration Action(s):
    • 2-state Offline/Online banner w/ "Go Online" and "Last Refresh" and a hard-refresh button, include
    • Interstitial sign-in pane between clicking the app icon and initial client view; Erik's ASCII sketch for reference
    • Remove "Unread" messages from initial view in Offline mode

General

  • Need messaging to confirm successful action completion, and where actions took place (local, server, both)
  • "Conversations" concept confusing (Note: bucket nested within Sources list, to separate newsroom-reciprocated correspondence from un-reciprocated submissions)
  • Timestamp(s) in messaging pane too small/pale
  • Discoverability issues with "Reply To" block
  • Pointers on message bubbles, confusing
  • Language: "Export" more functionally clear than "Save"
  • Attached file things generally clear, some bumps to smooth-out
  • Idea validated that "Pick things to delete" as a dropdown item on a single Source, would be useful.
  • Next Iteration Action(s):
    • Include toast (transient notification) messaging—initially, for "Delete" flow, and with explicit messaging
    • Update "Delete Source" language to "Delete Account"
    • Remove "Conversations" bucket (currently nested within Source's list)
    • Simplify read/unread paradigm, with removed "offline" state and "Conversations"
    • Change all uses of "Save" to "Export"
    • Add Download icon, and omit word "Decrypt" in initial view—but note decryption happening in-progress.
    • Include "Latest Action" verbiage on messaging pane timestamp, and improve visible discovery
    • Include big paper-airplane-like button in "Reply To" field
    • Kill the bubble pointers, make less bubbly-looking

Participant Recruiting

Planning

Checklist: 2.1 Iteration

  • Kickoff
    • Meet to revisit goals, methods, and timing
      • Shift goals for this iteration?
      • Things to do differently?
      • Different users to recruit?
  • Preparation Communication Train
    • New Moderator/Note-taker pairs? Anyone interested in observation?
    • Schedule session blocks
    • Outreach to prospective Participants
    • Add new participants to spreadsheets
    • GCal invites to Participants, Moderator(s), Note-taker(s), and optional Observer(s)
    • Follow-up Moderator introduction email
  • Session Preparation
    • Updates to Research Plan?
    • New Session Outline
    • Update Discussion Guide
    • New Materials Assembly (prototype, cards, etc)
    • Internal meeting to communicate prototype updates & new interest points w/ pairs
    • Per-user, customized/password-protected prototypes generated?
    • Conduct sessions
    • Post-session synthesis
  • Wrapping-Up
    • Complete findings/synthesis synopsis; post to wiki
    • Compose dev/design action-items; share-out where appropriate (as GitHub Issue or in UxD wiki)
    • Post Template Prototype to wiki
    • Post Discussion Guide, Research Plan, Session Outline to wiki
    • Everything above: final artifacts attached here as links?
    • Write/Send thank you notes to Participants
    • Get rid of this damn checklist
    • Present/debrief findings to team
    • Post findings to Gitter and Forums

The Checklist: 2.0 Study

  • Kickoff
    • Create wiki page
    • Meet to discuss goals, methods, and timing
      • One-off study, or a series?
      • Timing of synthesis/findings/iteration cycles
      • How many sessions per iteration: where will the value be?
  • Preparation Communication Train
    • Create Study Spreadsheet to track all participants
    • Internal outreach to establish Moderator/Note-taker pairs, and to solicit interest in optional observers
    • Schedule session blocks
    • Initial outreach to prospective Participants
    • GCal invites to Participants, Moderator(s), Note-taker(s), and optional Observer(s)
    • Follow-up Moderator introduction email
  • Session Preparation
    • Create wiki page
    • Begin Research Plan
    • Begin Session Outline
    • Begin Discussion Guide
    • Begin Materials Assembly (prototype, cards, etc)
    • Complete Plan & Outline
    • Internal session dry-run
    • Complete Discussion Guide + Template Prototype
    • Per-user, customized/password-protected prototypes generated?
    • Conduct sessions
    • Post-session synthesis
  • Wrapping-Up
    • Complete findings/synthesis synopsis; post to wiki
    • Compose dev/design action-items; share-out where appropriate (as GitHub Issue or in UxD wiki)
    • Post Template Prototype to wiki
    • Post Discussion Guide, Research Plan, Session Outline to wiki
    • Everything above: final artifacts attached here as links?
    • Write/Send thank you notes to Participants
    • Get rid of this damn checklist
    • Present/debrief findings to team
    • Post findings to Gitter and Forums

Who Uses SecureDrop?
Learn about SecureDrop's users!

Design

Contributors

Learn!

User Research

Et cetera

Clone this wiki locally