PHASE 3: Comment Period Research Plan

Open Comment Period - Is eRegulations notice & comment a success?

Questions to answer

External users - Frequent commenters and the general public

• Are commenters using eRegs Notice and Comment to submit comments to EPA, or are they sticking with what they are used to?
  • Why are they, or aren’t they using the tool?
  • If they are using the tool:
    • When in their commenting process are they using the tool?
    • How are they using the tool? Are they only using certain parts of the tool? (Just submitting comments? Just reading the reg?)
    • Do commenters want to use this tool to comment on future rules?
  • If they are not using the tool:
    • Is something missing that is impeding their commenting effort?

Internal users - Rule writers and contractors sorting comments

• Is it easier and faster to sort comments presented in this new format?
• What differences are see between the comments coming in for this rule and comments collected in previous rulemakings?
• Will this save them time and money? How much?

Variables

• Number of comments
  • We could get anywhere between 10 and 200.
• Timeline
  • Most commenters will most-likely submit their comments towards the end of the 60 day period, or at the very last minute.
• Drafting a comment can be more than just writing
  • Many frequent commenters are not commenting for themselves, meaning they need to get various levels of sign off from various people or groups of people before submitting their comments.

Types of research we can do

Interviews with users and internal staff

• External users -
  • Using ethn.io, we can catch people at the beginning of their process and at the very end. By sitting down and talking to people about their experiences before and after their comment is submitted, will help us to gain insight and answer some of the above questions.
• Internal users -
  • Ask someone sorting comments how the new format is working or not working for them.
• Interview discussion guides (Link TBD)

Usability testing

• External users -
  • If at all possible - Ask users to submit their comments while we watch.
  • More likely - After commenters have submitted their comments to EPA, in a user interview, ask users to walk us through how they submitted their comment.
• Internal users - Ask someone sorting comments to sort a few while we observe. (This can probably be done at the same time as the more general interview.)

Diary study

Goal - Catch commenters “in the moment” as opposed to before or after the full actions are complete.

• External users
  • 3-6 users
  • Request users in Bill’s initial outreach email
  • Once a day (or once/twice a week) email participant with 3-4 questions about what they did that day related to the comment until they submit their comment
  • Could be done all through email OR using google forms / survey monkey
• Internal users
  • 3-6 users
  • Users split between contractors and rule writing staff
  • Once a day email participants 3-4 questions about how many comments were sorted, how easy were they to sort, any issues? Quality of comments?

Research timeline

July XX, 2016 - Rule is published in the Federal Register and the comment period OPENS!

60 DAYS STARTS NOW!

Weeks 1-2 (first 14 days) - Gauge initial reaction

• Monitor Ethn.io screener to see if anyone responds
• Conduct interviews with up to 4 (?) external users (based on ehtn.io responses)
• If diary study - Set up users with diary study

Weeks 3-4 (28 days) - Monitor and wait

• Continue to monitor Ethn.io screeners
• If diary study - Continue to collect data from participants
• Maybe conduct a few usability tests in this time (most likely the task base tests as opposed to watching someone comment)

Weeks 5-6 (42 days) - Monitor and wait

• Continue to monitor Ethn.io screeners
• If anyone responds to the “success” ethn.io screener, begin user interviews with that group.
• If diary study - Continue to collect data from participants

Weeks 7-8 (56 days) - Monitor and wait

• Continue to monitor Ethn.io screeners
• User interviews with commenters who have sent their comments to EPA.
• User interviews with internal staff.
• If diary study - Continue to collect data from participants - if external users have completed their comment, start conducting interviews to review their experience

Weeks 9-10 (60 days and then some) - Final results

• Final interviews if people who completed their comment at the last minute (first few days)
• If diary study - Final interviews with diary study participants.
• SYNTHESIS and ANALYSIS and report creation

---

General background

• Phase 1: Generative research
  • Original research plan
  • Findings
  • Workshop with EPA results and user stories
• Phase 2: Testing the tool
  • Findings
• Phase 3: Success?
  • Comment period research plan < You are here!
  • Findings