TWE: IS21: RP001: intern Exit Interview Recordings: De-identify: Intern 005

[pandanista]

Dependencies

- [ ] All the user interview issues have been created IS21 didn't set up those issues, but all interviews were conducted. See TWE: IS21: RP001: Intern Exit Interview Roll Call and Session Table
- [ ] Interview Recording Upload Form needs to be created The interviews were conducted before we set up the upload form, so it doesn't apply in RP001's case

• Interview recordings have been uploaded and moved into Internship - PII's "My Drive"
• Interview transcripts have been created

Overview

We need to clean up and de-identify RP001's Intern Exit interview transcripts so that we can move on to the data analysis and insights generation phase.

During these tasks, you will be asked to add links to the Resources/Instructions section, and that section is at the bottom of this issue (go view it now).

Another tip is to use split browser windows side by side when working on any GitHub issue, so you can edit the issue on one window, and check the resources or click through links on the other window to avoid a lot of back and forth. Make sure to refresh both windows to reflect any changes you made, including updating the issue or checking the boxes on one of them.

Action Items

• UX lead adds the assignee to the Internship - PII's "My Drive" as a Contributor so that they have access to Internship - PII's My Drive where the recordings and transcripts with PII are stored.
  • UX lead accesses the Internship - PII's "My Drive"
  • Choose "Manage members", which is located towards the top right side of the browser
  • Enter the assignee's email address and select the role as "Contributor"
  • Confirm with the assignee that they have access to the Internship - PII's "My Drive"

Customize Resource Links

• Customize Resource for Wiki Page Link

  • Go to the wiki page: Research Output Overview (Resources # 1.01)
  • Find relevant wiki page for RP001
  • Copy the link for the wiki page
  • In Resources # 2.01, place the link you just copied between parentheses at the end of the line with no space in between the right bracket ] and the left parenthesis (, so it turns into a hyperlink
  • Choose "Update comment" in Github and make sure all the checkboxes above have been checked

• Customize Resource for this Research Plan's Google Drive Folder

  • Open the Google Drive's Research by Type Folder (Resources # 1.02)
  • Choose the Intern folder
  • Choose the RP001 folder
  • Copy the link of the RP001 folder
    1. Choose the three vertical dots on the right side of the RP001 folder
    2. Choose "Share"
    3. Choose "Get link"
    4. Choose "Copy Link" and "Done"
  • In Resources # 2.02, place the link you just copied between parentheses at the end of the line with no space in between the right bracket ] and the left parenthesis (, so it turns into a hyperlink
  • Choose "Update comment" in Github and make sure all the checkboxes above have been checked

• Customize Resource for Interview Recordings and Transcriptions Tracking Sheet

  • Open the Research Plan's Google Drive folder in the Internship's shared drive (Resource # 2.02)
  • Locate the TWE: IS21: RP001: Intern Exit Interview Recordings and Transcriptions Tracking Sheet in the folder
  • Copy the link of TWE: IS21: RP001: Intern Exit Interview Recordings and Transcriptions Tracking Sheet
    1. Choose the three vertical dots on the top right side of the file
    2. Choose "Share"
    3. Choose "Get link"
    4. Choose "Copy Link" and "Done"
  • In Resources # 2.03, place the link you just copied between parentheses at the end of the line with no space in between the right bracket ] and the left parenthesis (, so it turns into a hyperlink
  • Choose "Update comment" in Github and make sure all the checkboxes above have been checked

• Customize Resource for Interview Recording Folder stored in the Internship - PII's My Drive

  • Log into your Google account so you will be able to access the Internship - PII's 'My Drive' in the next steps
  • Choose Shared drives > Internship - PII > Internship -PII
  • Choose TWE Interview Recordings by Plan # folder in Internship - PII's My drive (Resources # 1.03)
  • Locate the video recording folder for RP001 inside the folder TWE Interview Recordings by Plan #
  • Copy the link of the video recording folder for RP001
    1. Choose the three vertical dots on the right side of the folder
    2. Choose "Share"
    3. Choose "Get link"
    4. Choose "Copy Link" and "Done"
  • In Resources # 2.04, place the link you just copied between parentheses at the end of the line with no space in between the right bracket ] and the left parenthesis (, so it turns into a hyperlink
  • Choose "Update comment" in Github and make sure all the checkboxes above have been checked

• Customize Resource for the De-identified Transcripts Folder in the Shared Drive 

  • Go to RP001 folder (Resources # 2.02)
  • Locate the RP001 De-identified Transcripts folder
  • Copy the link of the folder
    1. Choose the three vertical dots on the right side of the folder
    2. Choose "Share"
    3. Choose "Get link"
    4. Choose "Copy Link" and "Done"
  • In Resources # 2.05, place the link you just copied between parentheses at the end of the line with no space in between the right bracket ] and the left parenthesis (, so it turns into a hyperlink
  • Choose "Update comment" in GitHub and make sure all the checkboxes above have been checked

• Customize Resources for the Recording and Transcript you are Assigned to

  • Check the title of this issue to identify the participant number, which comes after De-identify:
  • Open the TWE: IS21: RP001: Intern Exit Interview Recordings and Transcriptions Tracking Sheet (Resources # 2.03)
  • Locate the recording video in .mp4 format that matches the participant number in Column B of the tracking sheet
  • Copy the link of the recording that matches the participant number in Column B of the tracking sheet
  • In Resources # 2.06, place the link you just copied between parentheses at the end of the line with no space in between the right bracket ] and the left parenthesis (, so it turns into a hyperlink
  • Locate the transcript in .txt format that matches the participant number in Column C of the tracking sheet
  • Copy the link of the .txt file that matches the participant number in Column C of the tracking sheet
  • In Resources # 2.07, place the link you just copied between parentheses at the end of the line with no space in between the right bracket ] and the left parenthesis (, so it turns into a hyperlink
  • Prepare for cleaning up and de-identification by converting the .txt file into a Google Doc
    1. Click on the link in Column C Transcription Link that matches the participant number (Resources # 2.07)
    2. Choose Open with Google Docs 
    3. A Google Doc is generated in a new window with the same .txt file name
    4. The Google Doc is now saved into the same folder with the corresponding video and .txt file
    5. Open the RP001 Interview Recording folder (Resources # 2.04)
    6. Open the corresponding folder for Intern 005 and check to make sure a Google Doc transcript has been created
  • Make a copy of the Google Doc that was just generated and rename it
    1. Choose the three vertical dots on the top right side of the file
    2. Choose "Make a copy"
    3. A copy of the Google Doc is generated in the same folder
    4. Copy

    To be de-identified-
    

    1. Go back to the folder where a copy of the Google doc was created
    2. Choose the three vertical dots on the top right side of the file
    3. Choose "Rename"
    4. Remove Copy of from the file name when the Rename text box pops up
    5. Paste what you copied into the beginning of the file name text box
    6. Choose "Ok"
  • Copy the link of the to-be de-identified Google Doc transcript you just created
    1. Choose the three vertical dots on the top right side of the file
    2. Choose "Share"
    3. Choose "Get link"
    4. Choose "Copy Link" and "Done"
  • In Resources # 2.08, place the link you just copied between parentheses at the end of the line with no space in between the right bracket ] and the left parenthesis (, so it turns into a hyperlink
  • Choose "Update comment" in Github and make sure all the checkboxes above have been checked

Clean up and de-identify the transcript

• Open the Google Doc transcript if it is not open yet (Resource # 2.08)
• Listen to the recording to get yourself familiar with the transcript
• Listen to the recording again and clean up the inaccuracies in the transcript as you read along
  • Separate the texts based on speakers and timestamps (timestamps should match the video file timestamps in case a researcher needs to go back and double-check the original video)
  • Add any missed words
  • Edit any typos and misidentified words
  • Deleted repeated words
  • Remove verbal filler words such as "um" and "uh" if they don't serve any purpose
    • Sometimes the verbal fillers can indicate the interviewee's emotions or thoughts. In this case, provide context in square brackets to demonstrate the emotions or thoughts (e.g., [The participant hesitated for a while before coming up with the answer]).
  • Where needed, add context in square brackets so that a reader could understand what was happening without watching the video (e.g., [steps off camera])
• Read through the transcripts again and make edits to de-identify any personal information
  • Search for the participant's name
  • Replace with their participant number, i.e. I001
  • Search for unknown speakers in the transcript
  • Replace unknown speaker with UXR number or other participant type/number
  • Read through the transcripts to search for any identifiable information in the interview
  • Replace any personal identifiable information with non-identifiable terms. For example, if someone's name is mentioned, we should use their roles inside square brackets so the reader knows this information was redacted (e.g., [team member], [peer], [mentor], [manager]). If their specific school is mentioned, we should redact that info with either [high school] or [college].
  • If they mentioned any specific issues they worked on, make sure to remove the issue numbers and rephrase the issues they worked on
• Read through the edited transcripts again
  • Focus on punctuation, readability, and formatting.
• When you're satisfied that the transcript is completely deidentified
  • Click on the version history and select the most current version
  • Click the three dots and select "Make a copy", and the "Copy version" window pops up
  • In the "Name" text box, remove the "Copy of" text at the beginning and the timestamp at the end from the file name
  • Copy

  De-identified Interview Transcript-
  

  • Paste the text you just copied in front of the file name in the "Name" text box
    • The new file name should be formatted like "De-identified Interview Transcript-RP001-UX Researcher Abbreviation###-Intern Abbreviation###"
    • An example: "De-identified Interview Transcript-RP006-U007-I001"
  • Save the renamed de-identified Google doc transcript into the RP001 De-identified Transcripts folder. This is to make sure that the version history does not include any identifiable information.
    1. Click the folder icon under "Folder"
    2. Choose "All locations"
    3. Choose Shared drives > Internship > Internships > Research > Research by Type > Intern > folder for RP001 > RP001 De-identified Transcripts
    4. Choose "Select"
    5. Select "Make a copy"
    6. The copy of the file is generated in a new browser
    7. Dismiss the notification that says the file is now in the shared drive
  • Copy the link of the de-identified transcript in the shared drive
    1. Choose "Share"
    2. Choose "Get link"
    3. Choose "Copy Link" and "Done"
  • Paste what you copied into the matching participant's cell in Column D Transcription Link - de-identified in the tracking sheet (Resources # 2.03)
  • In Resources # 2.09, place the link you just copied between parentheses at the end of the line with no space in between the right bracket ] and the left parenthesis (, so it turns into a hyperlink
  • Choose "Update comment" in GitHub and make sure all the checkboxes above have been checked

• Review with UX Lead
• Product sign-off
• UXR Lead or PM removes the assignee from the Internship - PII's My Drive when closing the issue if the assignee no longer needs to access the PII drive.

Resources/Instructions

Resources for creating this issue

1.01 Wiki: Research Output Overview
1.02 Google Drive Folder: Research by Type
1.03* TWE Interview Recordings by Plan #
1.04 Guidelines to Interviews Page 8
1.04.01 Brackets vs. Parentheses
1.04.02 Video folder
1.04.03 Cleaning Up Zoom Transcriptions for Qualitative Research
1.04.04 Determining Best Practice for Filler Words in Captions and Transcripts

Resources gathered during the completion of this issue

2.01 Wiki: Research Plan RP001
2.02 Google Drive Folder: RP001
2.03 TWE: IS21: RP001: Intern Exit Interview Recordings and Transcriptions Tracking Sheet
2.04* RP001 Interview Recording Folder
2.05 RP001 De-identified Transcripts
2.06 Intern 005 Recording Link
2.07 Intern 005 Transcript .txt File
2.08 Intern 005 To be De-identified Transcript Google Doc
2.09 Intern 005 De-identified Transcript Google Doc

*This folder can only be accessed from the Internship - PII's "My Drive"

[pandanista]

Hi @AndrewSalvatore, I assigned intern 005's transcript de-identification issue to you cuz I remember you were working on it.

Also, this issue is based on a new template, so you will need to gather the links again. It might be easier for you to copy from the resources section from #355.

For context, I've split the original transcribe template into two: one for transcribing and setting things up, and the other one is for de-identification.

Let me know if you have any questions.

[pandanista]

@AndrewSalvatore FYI: I've updated the issue above and created a to be de-identified Google doc in the intern 5 folder. Thank you for your feedback.

[AndrewSalvatore]

Progress: 25% done with transcript cleaning/deidentifying
Blockers: none
Availability: 6 hours
ETA: next Tuesday
Pictures or links (if necessary): none

[AndrewSalvatore]

Progress: 80% done with cleaning/deidentifying
Blockers: none
Availability: 6 hours
ETA: today
Pictures or links (if necessary): none

[pandanista]

@AndrewSalvatore Thank you for working on this issue. The transcript looks good. I made some minor edits to clarify a few sentences. It is ready for the Product to review now.

@sunannie27 Would you please review the deliverable in Resources 2.09? Thank you.

[sunannie27]

@pandanista We think it may be better if we color code the interviewer/interviewee portions or create an icon so they can be better distinguished.

[pandanista]

@sunannie27 We have decided not to color code the transcript for now since the analysis stage will involve color coding.

Drew has updated the transcript by incorporating the intern number, please see the updated de-identified transcript linked in Resources # 2.09. Let me know if there is anything else needed. Otherwise, we can close this issue after Product's final review. Thank you.

[pandanista]

We need to decide on the format of the timestamps (00:00:00 vs 00:00).

[pandanista]

@AndrewSalvatore Would you please take another look at the transcript?

We have decided at the all-team meeting today, that instead of Grammarly, please use LanguageTool Chrome Extension to clean up the punctuation and verbal ticks in the transcript. Thank you very much!

I've copied Resources 1.04 below for easy reference on how to clean up the interview transcripts.
1.04 Guidelines to Interviews Page 8
1.04.01 Brackets vs. Parentheses
1.04.02 Video folder
1.04.03 Cleaning Up Zoom Transcriptions for Qualitative Research
1.04.04 Determining Best Practice for Filler Words in Captions and Transcripts

[pandanista]

@AndrewSalvatore Welcome back, and hope your 2024 is off to a good start.

Would you please provide an update?

1. Progress: "What is the current status of your project? What have you completed and what is left to do?" - we know you just came back from break, please just summarize the progress from before then
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures or links* (if necessary): "Add any pictures or links that will help illustrate what you are working on."

• remember to add links to the top of the issue if they are going to be needed again.

[pandanista]

Added a few guidelines on the transcript formatting. You are already following most of them. I am listing them below as a reference.

1. Use the 'in-line' format as seen here in Intern 001's transcript
2. Use 6 digit format for all the timestamps, for example, 01:01:01 as hour one/minute one/second one
3. Use "Interviewer" and "Intern ###" throughout the transcript. See an example here in Intern 001's transcript.
   • If there were more than two people recorded on the transcript, name the third person based on the interview set-up or context, such as Program Manager, Note Taker, etc. If you are not sure what to name them, ask leads on the project to clarify.
4. If there are other numerical values in the transcript, transcribe them using texts. For example, if a time such as 6:30 is mentioned, type out six thirty when cleaning up the transcript. Another example is if the interviewee mentioned he mainly sought help from 3 other peers, type out three instead of using the number 3.

[sunannie27]

@AndrewSalvatore
Please provide update

1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures or links* (if necessary): "Add any pictures or links that will help illustrate what you are working on."

• remember to add links to the top of the issue if they are going to be needed again.

[pandanista]

@AndrewSalvatore Hi Drew, would you please provide an update on this transcript?

I included some more instructions and resources in comment 1 and comment 2 above so we can stay consistent when cleaning up the transcript. Let me know if you have any questions.

[AndrewSalvatore]

Progress: Reviewed the transcript and it looks like the formatting abides by all the rules mentioned in @pandanista's comments
Blockers: none
Availability: 6 hours
ETA: done
Pictures or links (if necessary): none

[pandanista]

Awesome. Thank you, @AndrewSalvatore.

@sunannie27 I reviewed the transcript, and it looks good to me. Would you please review the transcript in Resource 2.09 before we close the issue? Thank you!

[sunannie27]

@AndrewSalvatore
We noticed this large paragraph of text in the transcript and thought there should be a paragraph break before "As for help". If there is a lot of talking by one person and there are places that seem logical for a separate paragraph tag, then we should insert them for it to be easier to process.

Paragraph without breaks (currently in the document)

There weren't really any issues in Java. I think there's only one team in Hack for LA that really uses Java, and also because almost all my knowledge in Java was mostly just from school I never really… I mean I'd made programs obviously for school work but never actually worked with it in a professional setting before. I don't really know what motivated me but I think it was because I had to learn so I did learn. As for help, what I did at first I usually just Googled to see if I could just find something that explained to me how to fix the problem, or what to do. Because also usually with the issue they usually gave a pretty good description of what you needed to do and also they usually give you resources to check out so that you could check on the things that they thought that you would probably need to use to complete the issue. If I couldn't figure it out that way, I'd usually just reach out on the Hack for LA site channel or there were some other people I would just direct message and ask like “Hey, can you help me?” And then for one of the issues, it was for… because my first large issue was also my first back-end issue I also got a couple of mentors to help me out. Oh, actually no. Really only one [mentor].

Paragraph with breaks

There weren't really any issues in Java. I think there's only one team in Hack for LA that really uses Java, and also because almost all my knowledge in Java was mostly just from school I never really… I mean I'd made programs obviously for school work but never actually worked with it in a professional setting before. I don't really know what motivated me but I think it was because I had to learn so I did learn.

As for help, what I did at first I usually just Googled to see if I could just find something that explained to me how to fix the problem, or what to do. Because also usually with the issue they usually gave a pretty good description of what you needed to do and also they usually give you resources to check out so that you could check on the things that they thought that you would probably need to use to complete the issue.

If I couldn't figure it out that way, I'd usually just reach out on the Hack for LA site channel or there were some other people I would just direct message and ask like “Hey, can you help me?” And then for one of the issues, it was for… because my first large issue was also my first back-end issue I also got a couple of mentors to help me out. Oh, actually no. Really only one [mentor].

[sunannie27]

There is a break used with hyphens but hyphens should only be used if there is an emphasis on what was said in the interview, otherwise use commas.

I really appreciate you sharing your honest feedback, and your responses were very insightful. So thank you so much for that. I really appreciate you taking the time to share your journey, your experience, your hiccups, and everything else. Thank you. Getting feedback from you - as I mentioned before - is really going to help us improve and enhance the program for our prospective interns. And hopefully it'll enable students interested in STEM programs to learn and move forward with confidence in their workspace. So, I really appreciate your time, and it looks like we're pretty motivated!

[AndrewSalvatore]

Thank you for being so patient with us, @AndrewSalvatore!

Would you please update "interviewer" with "UXR 001" in intern 005's transcript? Much appreciated!

We decided to include the UXR numbers in the transcripts because 1). there might be different probing questions for semi-structured interviews from different interviewers, and 2). it will be easier to reference the UXR when doing analysis or making highlight reels (if needed).

@pandanista Done

[pandanista]

Thank you very much, @AndrewSalvatore. I am closing this issue since it is all done.

[pandanista]

• Reopening this issue since we need to track people being mentioned in the transcripts.

• UX Research Lead assigns the assignee to the PII drive so they have access to the recording and original transcripts

• Fill out any individual's name and relevant information in the TWE: IS21: RP001: Participants List from Interviews (Interns & UXR Excluded) spreadsheet by following the steps below:

  • Open the TWE: IS21: RP001: Participants List from Interviews (Interns & UXR Excluded) spreadsheet in a new tab
  • Scroll up to the Resources section to locate recording video and transcripts. They are located at the end of the first comment of this issue. See screenshot.
  • Listen to the interview recording in Resources 2.06 to get familiar with the content
  • Read through the transcript(s) in Resources 2.08 and Resources 2.09 to identify any individual that was mentioned by the intern during the interview
    • Their name in the transcripts might have been replaced by their participant type or number or even spelled wrong, so pay close attention to the transcripts.
  • Listen to the recording in Resources 2.06 again while reading through the transcript(s) in Resources 2.08 and Resources 2.09
  • Identify any individual's name being mentioned, and list out all the relevant information in the spreadsheet. No need to include intern and the UXR names since we track them in the roll call and session table.
    • Please check each participant type's abbreviation under the "Research Documents by Participant Type" section of the Research Output Overview Wiki Page. For example, mentor is M.
    • The participant numbers are sequential based on the time of entry and occurrence of the mention. I.e. if there are already two Hack for LA website team members being listed previously in the spreadsheet, then the next Hack for LA website team member will be HfLAWTM003.
    • If the name is already included in the sheet because other interns have mentioned them, please still list out their names and relavent informatioon. Also, for the people who were mentioned repeatedly, please make sure to use the same role and the same participant number that are already assigned to them so we can keep track of the people being mentioned across all interviews.
    • If you are not sure of the roles of any of the names mentioned, please list the names, timestamps, and transcript links in the spreadsheet, and leave the role column and the participant number column blank.
      • Leave a comment to ask for clarification in the transcript by highlighting the individual's name.
      • Leave a new comment in this issue or/and inform the UXR lead at the next meeting
      • UXR lead confirms the role of the individual
  • After identifying and confirming on the names, roles, and other relevant information, use the search and replace function in Google Doc to replace any identifiable names in the de-identified transcript (link in Resources 2.09) with the participant number assigned to them from the spreadsheet.

• UXR Lead will review the TWE: IS21: RP001: Participants List from Interviews (Interns & UXR Excluded) spreadsheet

• UXR Lead will add the de-identified participants' information to the TWE: IS21: RP001: De-identified Participants List from Interviews (Interns & UXR Excluded) and save in RP001's folder in the shared Google Drive

Let me know if you have any questions. Thank you.

[pandanista]

@floydns: I am assigning this issue to you.

• Please listen to the interview recording in Resources 2.06, and identify any individuals that were mentioned in the interview session with Intern 002 in the transcripts (Resources 2.08 and 2.09) by listing out their info in a spreadsheet.
  • The instructions can be found in this comment.
  • The names might already be de-identified. For example, project lead or lead mentor might be used in the de-identified transcript. In that case, list out the name (based on the interview recording), role, and other info in the spreadsheet. If the name wasn't there in the spreadsheet, assign them a new participant number based on participant type. If the name already exists in the spreadsheet, then use the same participant number they are already assigned to.
  • Please check off the boxes in the comment above as you go along.

An example of a similar issue can be found here.

Thank you very much!

[floydns]

@pandanista I believe I can finish this over the next few weeks. Just slammed at the moment :)

[pandanista]

@floydns Thank you for the update. Let me know if you have any questions.

[pandanista]

I am re-assigning the issue to @jinyan0425.

• Please listen to the interview recording in Resources 2.06, and identify any individuals that were mentioned in the interview session with Intern 002 in the transcripts (Resources 2.08 and 2.09) by listing out their info in a spreadsheet.
  • The instructions can be found in this comment.
  • The names might already be de-identified. For example, project lead or lead mentor might be used in the de-identified transcript. In that case, list out the name (based on the interview recording), role, and other info in the spreadsheet. If the name wasn't there in the spreadsheet, assign them a new participant number based on participant type. If the name already exists in the spreadsheet, then use the same participant number they are already assigned to.
  • Please check off the boxes in the comment above as you go along.

An example of a similar issue can be found here.

Thank you very much!

[jinyan0425]

Hi Ying. It is done and I left some small comments in the resource 2.09 Intern 005 De-identified Transcript Google Doc @pandanista Please review the relevant documents and let me know if you have any questions. Thanks!

[pandanista]

@jinyan0425: Thank you for highlighting and commenting on the doc. I have reviewed them, and updated or corrected developers' names in TWE: IS21: RP001: Participants List from Interviews (Interns & UXR Excluded) spreadsheet.

Everything looks great.

I have also added the relevant info to TWE: IS21: RP001: De-identified Participants List from Interviews (Interns & UXR Excluded).

We can close this issue now.