Design a survey for new volunteers for CTJ metrics

[stephaniestahlberg]

Overview

As researchers, we need to create a survey for new volunteers so they can evaluate the Hack for LA process for joining teams. These survey responses will generate scores on the CTJ metrics we have defined and can be compared to how people evaluate the joining process using CTJ in the future.

Action Items

• Brainstorm ideas for collecting data on metrics
• Write a research plan
• Design the survey
• Finalize CTJ metrics for new volunteers with PMs/team
• Make sure that survey questions would serve as appropriate metrics for both the old and new processes
• Discuss the draft survey with the team and HfLA org
• Edit the survey accordingly
• Test the survey with 2 participants
• Make any final edits needed
• Send the survey to target respondents

Resources/Instructions

Research Plan
Survey in Google Forms

[MariaElizabeth04]

Here is the link for the current version of the survey https://forms.gle/PaGERmAthzo3RMUEA

[MariaElizabeth04]

@sdimran @sharadgaurav7 This issue has been updated. The research plan and survey have been created and added to resources for you to review and comment on.

[sdimran]

some notes :
in Q1

• the first and last statement are the opposite, so would answering one of them not answer the other? ex. if someone's response was "Strongly agree" for "The steps for joining a project were clear to me." then wouldn't that mean that their response for "Parts of the project joining process were confusing to me." would be "Strongly disagree"? do we need to have both questions

in Q2

• can we make the phrasing "Rate the level of effort it took for you to join a project from 1-5"

we can review with bonnie on sunday

[stephaniestahlberg]

Thank you @sdimran. I incorporated your suggestions, and also added a screener question to check that respondents have joined a project in the last 6 months, and therefore are more likely to remember the process well. It's ready for Bonnie's review tomorrow.

[stephaniestahlberg]

We reviewed this with Bonnie today and it's good to go. The only change to be made is to remind respondents in question 6 that we're asking about their experience after onboarding. @MariaElizabeth04 and I will test the survey with 2 participants this week, and when it's ready Bonnie said she can personally post this on Slack channels to elicit more responses.

[MariaElizabeth04]

This survey has been sent to Bonnie and will now be sent to participants!

[MariaElizabeth04]

Some of the feedback that we received after testing it with 3 participants:

Participant 1- Recently joined Hack for LA (within 3 months): Found the survey to be clear and short enough, though some questions seem too similar to them.

Participant 2- Recently joined Hack for LA (within 3 months):
-Liked the anonymity of it.
-"The postings for Open roles at Hack for LA are easy to find" question. Participant shared that the postings for role are easy to find if you save/bookmark the link shared at onboarding. Maybe ask if easy to refer back to, or find again. Maybe add a screener asking if they have bookmarked it, and how they found it. Maybe can cross-reference with whether people bookmarked the page.
Question 7: a 10 rating based on tools and skills learned, but an 8 based on time and energy commitment. Based on schedule, pace, and how hectic it is.
Based on this feedback for question 7, we decided to make question 8 required so we can better understand why people give a certain rating.
-Adding "Me" to question number 3 of the survey to make sure that the question addresses the participant directly and more specifically.

Participant 3- Recently joined Hack for LA (within 6 months):
Wonder about the meaning of onboarding. Feedback wasn't as user perspective-based, but more for what was the reasoning behind the questions themselves.

Steps: We cut out one of the questions from section number 3, to make it less repetitive, and remove the requirement from question number 8, which we had made after meeting with participant number 2.