TWE: Create Research Plan for RP017

[bradyse]

Dependencies

• TWE: Create Roadmap: Intern Research #309
• A research setup structure (wiki, GitHub labels, Google Drive, etc.) has been created. TWE: Research Structure for RP017: Productivity Dashboard Interviews #291

Overview

We need to create a Research Plan for Productivity Dashboard Interviews so that we can understand interns' experience with the productivity tracker.

Action Items

• Update this issue to make sure it's properly categorized and easy to manage
  • Under "Projects," add to the Project Management Board (helps with Project Management overview)
  • Add the milestone: Research Plan Creation (helps with prioritization)
  • Add label of the correct research plan number associated with this research plan (e.g., Research: RP001) (helps you find related issues in the same workflow)
  • Edit title of issue with the correct research plan number (same as prior)
  • Edit placeholders in Overview of this issue with the relevant information in square brackets. (personalizes the issue to make it easier to read). Any place you see [replace...] replace both the text and the brackets with what it is asking for (including in the front matter above but not where it says name, but not where it says title.
• Update this issue with the relevant Resources:
  • Go to the Research Output Overview Page (link in Resources)
    • find the relevant overview type wiki page.
    • change [Wiki [TYPE] Research Overview Page] in the Resources below with the correct wiki page of the research overview by type.
  • Go to the Research Output Overview Page again
    • find the relevant wiki page for this specific research plan.
    • change the [RP__ Wiki Page] in the Resources below with this RP's specific wiki page.
  • Find and link the folder you are going to put your research plan in (e.g., Research by Type>2. Intern>RP6 - Intern Intake Interviews)
    • Go to the Google Drive's Research by Type Folder (link in Resources)
    • Go into the relevant Type Folder (e.g., 1. Mentor, 2. Intern, etc.)
    • once inside, right-click on the relevant destination folder and copy the link. (e.g., RP6 - Intern Intake Interviews)
    • Add the Google Drive link in the Resources below with the link you just copied and revise the display text to include the Research Plan number (e.g., Google Drive RP__ Folder to Google Drive RP008 Folder)
• Create the Research Plan Document.
  • Make a new document using the TWE Research Plan template (link in Resources). Follow the instructions in the How To Write a Research Plan guide (link in Resources) and mark your progress in this issue.
  • Move your document from "My Drive" to the shared drive internship folder of this specific RP.
  • Name your document to be called "TWE: RP___ : [TITLE OF RESEARCH] Research Plan".
  • Update the document with the relevant information in accordance with the research roadmap document (found on the Research Overview Page).
    • Clean up transcript
    • Move de-identified transcript into RP project's Google Drive folder
    • Document all questions and follow-up questions by interviewer
    • Identify protocol based upon research questions
    • Summarize transcript for basic themes
    • Write data analysis for identification of basic themes
  • Update the Table of contents (page 2) when you are finished with the document.
• Meet and sign-off research plan document with UX Research lead (by adding it to the research team meeting agenda).
• Update RP Wiki Page with any changes or corrections.
• Once research lead has signed off, add it to the PM/Research lead agenda
• sign-off by PM
• Does this issue have any dependency checkboxes in the action items here?
  • if yes, remove dependencies that are listed below (issues that are waiting on this issue to be completed. Sometimes they have already been made and sometimes they yet to be made).
    • close this issue
  • if no, check to see if the follow-up issues have already been made (see related structure issue). If new issues have not been made, then apply label ready for research lead and define what needs to be done in a comment below
    • close the issue
    • move to Question/Review Column

Resources

Resources needed to complete issue

• Wiki Research Output Overview Page
• Google Drive Research by Type Folder
• TWE: Research Plan Template
• TWE: How to Write a Research Plan

Resources gathered during the completion of the issue

• Wiki Intern Research Overview Page
• RP017 Wiki Page
• Google Drive RP017 Folder
• RP017 Research Plan
• RP017 De-identified Transcript

[bradyse]

Note that research plan needs to be adjusted based upon one interview. See comment (and decision) made here.

[pandanista]

We don't have defined KPIs listed for the internship, but we have a couple of examples. The KPI examples can be found in

• RP001's research plan, which has listed KPIs approved by product.
• RP002's research plan also has KPIs listed.

Hope this helps with the KPIs.

Since we did one interview, I think we could align the product goals based on the interview transcript.

[bradyse]

1. Progress: In process of cleaning up original transcript. It is not fully deidentified yet. Because of this, it's still in the "My Drive" of the internship account.
2. Blockers: No blockers, but the transcript is slower going than I thought, because Bonnie didn't turn on the transcription until midway through the interview. So I am transcribing it by hand.
3. Availability: 1-2 hours
4. ETA: 2 weeks
5. Pictures or links (if necessary):

* remember to add links to the top of the issue if they are going to be needed again.

[pandanista]

1. Progress: In process of cleaning up original transcript. It is not fully deidentified yet. Because of this, it's still in the "My Drive" of the internship account. 2. Blockers: No blockers, but the transcript is slower going than I thought, because Bonnie didn't turn on the transcription until midway through the interview. So I am transcribing it by hand. 3. Availability: 1-2 hours 4. ETA: 2 weeks 5. Pictures or links (if necessary):

* remember to add links to the top of the issue if they are going to be needed again.

Thank you for the update, @bradyse. Transcribing def. takes time since we don't have a paid otter.ai account. But Otter also allows 3 free transcripts lifetime if you register for a personal account, which might be worth it.

[bradyse]

Thank you for the update, @bradyse. Transcribing def. takes time since we don't have a paid otter.ai account. But Otter also allows 3 free transcripts lifetime if you register for a personal account, which might be worth it.

1. Progress: Made a lot of progress. I don't think I need Otter, as I'm now cleaning up the Zoom transcript.
2. Blockers: No blockers but it just is taking a bit more time. Will try to work on it while on the road traveling this weekend.
3. Availability: 1-2 hours
4. ETA: 1 week
5. Pictures or links (if necessary): Link to in-progress transcript.

[pandanista]

@bradyse I just had a thought, and I wonder if you could help.

Are you doing de-identification after or along the transcript cleaning up process?

If yes, would you please write down the steps you took to de-identify the participant? So I could compare to what we have on the WIP transcribe template and incorporate some changes if needed.

Let me know what you think. Thank you!

[bradyse]

Are you doing de-identification after or along the transcript cleaning up process?

If yes, would you please write down the steps you took to de-identify the participant? So I could compare to what we have on the WIP transcribe template and incorporate some changes if needed.

Hi @pandanista, I just finished cleaning up the transcript and moved the de-identified copy into this RP's research folder. Here were the main steps I took:

1. Create a branded document with the raw transcript copied and pasted into the document.
2. Save this file inside the PII folder along with the video or other transcript files.
3. Clean up transcript:
   • Separate text according to speakers and timestamps (timestamps should match the video file timestamps so that a researcher needs to go back and double check the original video)
   • Add missed words
   • Change misidentified words
   • Deleted repeated words
   • Remove filler words such as "um" and "uh"
   • Where needed, add context in square brackets so that a reader could understand what was happening without watching the video (e.g., [steps off camera])
4. De-identify the research participant or any names mentioned between the interviewee and interviewer. (Note, I did not de-identify the interviewer)
   • Replace the interviewee's name inside the transcript with the participant type (e.g., "Intern")
   • Replace mentioned people in the transcript according to their roles inside square brackets so the reader knows this information was redacted (e.g., "[team member]".
   • If the folder where the video is located has the interviewee's name, rename the folder to replace their name with the participant type
5. Add relevant details to the transcript file regarding the date of interview and include the link to where the original video is.
6. When you're satisfied that the transcript is completely deidentified:
   • Click on the version history and select the most current version.
   • Click the three dots and select "Make a copy"
   • Save a copy into the shared drive folder where you want to save the de-identified version. This is to make sure that the version history does not include any identifiable information.
   • In the copied file, rename the file name to remove the "Copy of" text and the time stamp text.

[pandanista]

Thank you for sharing the progress and the steps, @bradyse.

[bradyse]

@pandanista

1. Progress: The RP017 Research Plan is ready for review by UX Lead and Product.
2. Blockers: None
3. Availability: 1-2 hours/week
4. ETA: I can meet next week as needed
5. Pictures or links (if necessary): RP017 Research Plan

[pandanista]

@bradyse Thank you very much for all your efforts! Has Bonnie provided any links to TDM usability tests yet?

@adarosh As a follow-up, please use the action items listed below as a to-do list for RP017's peer review.

Action Items for Peer Review

• Review this issue
  • Read the issue overview to learn about why this document was created.
  • Locate draft of the document or spreadsheet in the Resources.
• Review draft
  • Preview the draft quickly to get a sense of how the draft is structured and the type of content.
  • Review the structure of the draft to ensure that the headings (in a Google Doc) or columns (Google Sheet) are clearly identified.
  • Determine if the information is communicated in a way that is concise, precise, and free from jargon.
  • Identify whether the draft adheres to the basic branding guidelines for the project (see Branding or Design wiki link in Resources).
• Provide feedback
  • Add feedback in the form of (1) questions and/or (2) explicit instructions for how the content should be improved. Feedback in a Google Doc should be given in the form of comments.
  • If there are edits to be made, please use "suggesting mode" in Google doc instead of "editing mode".
  • Determine the accuracy of the content for each section (Google doc) or cell ( Google sheet). If you find an inaccuracy, error, or mis-categorization, explain how the information presented could be improved.
  • For each suggested improvement, provide a link or reference within your comment to justify your suggestion (if available).
• Specific to this issue:
  • Review documents linked in the comments in this issue to gather more information
  • Review Wiki Intern Research Overview Page
  • Review RP017 Wiki Page
  • Log into the [email protected] Google account and go to drive.google.com under that account
  • Go to shared drives > Internship - PII > Internship - PII > TWE Interview Recordings by Plan # > RP017 folder > recording with Intern about the dashboard 2022-07-13 folder
  • Watch the interview at 1.5X or 2X speed
  • Review the de-identified transcript to see if it needs any editing to provide more context or clarifications and/or redact any information sensitive
  • Review TDM's usability testing example (TDM Calculator has done usability tests that might be relevant to the productivity tracker interview)
  • Please provide concrete feedback based on the reviews you have done above in the Research Plan draft.
• Once all the steps have been completed, reach out to the original owner of this issue to determine if they can complete the next steps in the issue.

Thank you both!

Edited: Corrected the project name from VRMS to TDM. Added the TDM usability example and project card URL to the instruction section above.

[bradyse]

@bradyse Thank you very much for all your efforts! Has Bonnie provided any links to VRMS usability tests yet?

@pandanista No, she hasn't provided any links to my knowledge. I was a bit unsure what exactly she wanted me to do with the Transcripts but I do know adding screenshots would be helpful. It also would be just as helpful if whoever was doing the analysis just had access to the recording itself.

[sunannie27]

@adarosh
Please provide update

1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures or links* (if necessary): "Add any pictures or links that will help illustrate what you are working on."

• remember to add links to the top of the issue if they are going to be needed again.

[adarosh]

@pandanista @sunannie27 - The team lead Ying and I went through the document together last week (1/30) to address comments. For the few comments that needed clarification, Ying will discuss with team before we finalize.

[pandanista]

@sunannie27 We have addressed all the comments and made edits in the RP017 Research Plan. Please let us know if you have any other questions before the final approval.

[ExperimentsInHonesty]

@adarosh I have reviewed RP017 Research Plan and made a few grammar changes and suggestions.

Additionally, the sanitized version of the spreadsheet is linked to the research plan, but it would be good if we linked the actual one with a note saying it has restricted access because it contains pii

[adarosh]

Discussed research plan in UXR meeting on 3/12. Reviewed and made minor edit to paragraph addition in Background section. @pandanista will add action item spreadsheet.

[pandanista]

• Added the following assets to the resources section of the research plan document.
  • The action steps spreadsheet template
  • The action steps spreadsheet examples
  • The case study report template
  • The OG intern activity spreadsheet

• Made a note on the intern activity spreadsheet linked in the research plan body is de-identified, and the OG version is saved on the PII drive

• A new timeline with start dates and duration for future research is added.

@sunannie27 We are ready to close this issue since we have addressed all the comments left by Product. Please see the updated research plan document. Let me know if there is anything else.

[sunannie27]

@pandanista Looks good to me.

[ExperimentsInHonesty]

@adarosh Thank you for the work on this. We are closing this issue.

[mayankt153]

• All the links in this issue will need to be checked to see if we have access
• TWE: Create Roadmap: Intern Research #309
  • Have access, its located in ____
  • public
• TWE: Research Structure for RP017: Productivity Dashboard Interviews #291
  • I do not have access
  • public
• Research Plan Creation
  • Have access, its located in ____
  • public
• Wiki Research Output Overview Page
  • Have access, its located in ____
  • public
• Google Drive Research by Type Folder
  • need access, located in Internship Google Drive > Internships > Research > 1. Research by Participant Type
  • public
• TWE: Research Plan Template
  • I Have access
  • public
• TWE: How to Write a Research Plan
  • Have access, its located in ____
  • public
• Wiki Intern Research Overview Page
  • Have access, its located in ____
  • public
• RP017 Wiki Page
  • I do not have access
  • public
• Google Drive RP017 Folder
  • need access, located in Internship Google Drive > Internships > Research > 1. Research by Participant Type > 2. Intern > RP017 - Intern Activity Dashboard Interview
  • public
• TWE: RP017: Intern Activity Dashboard Interview Research Plan
  • need access, located in Internship Google Drive > Internships > Research > 1. Research by Participant Type > 2. Intern > RP017 - Intern Activity Dashboard Interview
  • public
• TWE: IS22: RP017: De-identified Interview Transcript
  • need access, located in Internship Google Drive > Internships > Research > 1. Research by Participant Type > 2. Intern > RP017 - Intern Activity Dashboard Interview - still needs pii removed
  • public