Design a rating system for interview preparedness

[ExperimentsInHonesty]

Dependencies

• IS22: Interview Questions #138
• UX lead needs to ask admin to add the assignee to the internship UI/UX vault

Overview

We need a rating system for interview preparedness so that we can measure the improvement in their preparedness based on our coaching.

Action Items

• Identify which recordings we should have
• Find location of the recordings and add that link to resources below
• If we are missing recordings reach out to Bonnie by adding it to the team meeting agenda
• Make two copies of google doc using the TWE: Interview Summary Template linked below (Resources 1.03)
• For the first copy, copy

TWE: IS22: RP023: Intern Application Interview Summary

• Paste to replace the title of the cover page of the google doc
• Paste into the top field to rename the file
• Move the google doc to the shared drive > Internships > Internships > Research > Research by Participant Type > Intern > RP023 - Intern Application Interview
• For the second copy, copy

TWE: IS21: RP023: Intern Application Interview Summary

• Paste to replace the title of the cover page of the google doc
• Paste into the top field to rename the file
• Move the google doc to the shared drive > Internships > Internships > Research > Research by Participant Type > Intern > RP023 - Intern Application Interview
• Review the recordings from the interviews to identify all the pitfalls by listening to the notes we gave after the interviewee left
• Gather notes and transcripts from the end of each video and add them to the interview summary google docs
  • There were not physical notes in the drive regarding interns' interview performance. However, interviewer kept notes during the interviews and the notes were verbally discussed during the post application interview sessions. Make sure to listen to the end of the video to gather the notes.
• Compile a list of all of them and figure out groupings
• Design rating system

Resources/Instructions

1.01 Example of rating system from linkedin deck
1.02 Interview recordings: IS22 Candidate Application Interviews; IS21 Candidate Application Interviews
1.03 TWE: Interview Summary Template

[pandanista]

@TobyShanti The example of the rating system (see screenshot below from the deck) is not a good one. Let's keep it as a 1-5 or 1-7 Likert scale. Thank you!

[cbutcher318]

@TobyShanti Please provide update when you can.

1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."

[TobyShanti]

Progress: Roll call tables have been added. I need to identify which recordings we have by cross referencing the roll call table and where the recordings are housed. I need to also review the recordings to identify pitfalls and compile a list of them and design the rating system.
Blockers: I was waiting for access to the recordings so I can move to the next step, but I have access now.
Availability: I have Friday and Sunday to work on this issue.
ETA: I am not sure how long the recordings are, so I cannot provide an ETA at this time

[TobyShanti]

Availability for this week: 6 hours

Provide Update

1. Progress: I am reviewing the interview recordings. I am 1/4 through the videos.
2. Blockers: There were no notes at the end of the recordings for pitfalls, so I'm reviewing each of the videos for that information and putting together the findings. Videos range from 20 mins to 30 mins so taking some time.
3. Availability: 6 hours
4. ETA: I should have the videos reviewed by 3/7. I would like group input before creating the rating system after we review the findings.

**Please note that interviews for interns 5 and 8 are not in the "Intern Exit Interviews" folder, but were found in the "TWE: IS22: RP012: Intern Exit Interview Roll Call and Session Table" column "Recording link"

[TobyShanti]

Availability for this week: 6 hours

Provide Update

Progress: I have completed reviewing the interviews.
Blockers: Interviews were long with no notes.
Availability: 6 hours
ETA: I checked in with Ying and there's some clarification needed on how to create the likert scale. Ying is going to check in on 3/8's meeting to get better direction.

[pandanista]

For next steps, we need to confirm:

1. Is the rating system for interview preparedness used by UXRs on the TWE team or is it a self assessment done by interns themselves?
2. To measure the improvement, are we implementing a longitudinal study using repeated measure? For example, one rating at the beginning of internship application, another half way through and a final one at their exit.
3. Did we have any notes during interviews?
4. Would it make more sense to do this after we conduct data analysis for RP006 and RP012?
5. The structure of the rating

[priyatalwar]

@pandanista We discussed most of these today during our team meeting:

1. The rating system is for the interns themselves so they can self-assess
2. Yes
3. As discussed, you may be able to find notes on the issues themselves but we don't have a designated place for notes yet. Good idea to have one going forward.
4. We need this for our future cohorts , please post this question in the question and review column so we can review with Bonnie in 2 weeks
5. Can you elaborate on what do you mean by structure?

[pandanista]

@priyatalwar Thank you for answering them. I will list the questions above to the Q&A issue and double check with Bonnie after she is back.

[TobyShanti]

Availability for this week: 3 hours

Progress: I've started drafting a survey for the end of the internship.
Blockers: Interviews were long with no notes.
Availability: 3 hours
ETA: Awaiting check in with Bonnie for direction.

[TobyShanti]

I wanted to create a series of questions that helped us understand participant's skill levels walking into the internship, their expectations of the internship, their goals, and how they felt upon completing the program.

[pandanista]

Q1: Who is the intended user for the rating system for interview preparedness?

• More specifically, is the rating system going to be used by UXRs on the TWE team to rate how prepared each intern is during their intake and exit interviews? Or is the rating system used as a self assessment done by interns themselves?
  A1: The rating system is for TWE team to rate how prepared each intern is during their application interviews.

Q2: To measure the improvement, are we implementing a longitudinal study using repeated measures?

• For example, using the same rating across multiple touchpoints, such as the beginning of internship application, half way through, and their exits.
  A2: The rating system will be implemented during interns' workshops for targeted interns that needs to improve their interview performance. Therefore, it is not a repeated measure for all interns.

Q3: Did we take any notes during interviews?

• This refers to all the interviews we conducted in the past. If we took notes, where were the notes kept?
  A3: Informal notes were taken personally and discussed among the interviewers during the recorded sessions towards the end of application interviews.

Q4: Would it make more sense to do this after we conduct data analysis for RP006 and RP012 since we will have the transcripts?
A4: The rating system is derived from the application interview, not intake or exit interviews. So RP006 and RP012 are not relevant to this issue.

Q5: In terms of the structure of the rating system, are we looking for something similar to the screenshot below?

A5: We are planning to use checklist to rate the interns' soft skills and culture fit during the application interview.