Divide and conquer synthesis

[kelenelee]

BACKGROUND

The previous research team members completed affinity diagrams for the interview study, but not all the stickies are up to affinity diagramming standards.

I have directed the current research team to redo analysis, so that synthesis is more successful, so that we can extract meaningful, helpful, and actionable recommendations for HfLA’s practices as a whole.

In order to create a new, better analysis strategy, I had to rewrite the original research plan into a second research plan. The original plan's objectives and directives were too unclear that the other researchers could not effectively read and absorb its importance. It's important that the researchers read the new plan so that the study background is clear to everyone.

change log

• 5/24 @hdchangie @LeslieCi @allthatyazz @kelenelee discussed this task during the research team meeting
• 5/26 @hannahcarlan and @kelenelee had a Slack chat to clarify the action items and research goals. The research objectives were not written clearly in the plan. The plan has been updated with clearer research objectives that can translate into themes for analysis. In order to preserve the old plan, I decided to make a new doc instead of editing the original one.
  • As for analysis methodology - I was worried about asking the researchers to do the affinity diagramming technique. We are not trained in it by an expert and poor QC is why we have to redo it. But, because synthesis down the line will be difficult if everyone's analysis methods are NOT the same, I am asking that everyone watch a video about affinity diagram best practices from a Delta CX tutorial.
• 5/31 @kelenelee updated the GitHub issue to clarify new updates from 5/26. Based on the current state of progress, I don't think it's wise to pivot back to the Figma Kit by 6/6. The researchers have put in a lot of time to rewatching the interview videos, and if we take a break now before we complete synthesis, then we may have to do repeat work whenever we come back to this study. Furthermore, if there are findings that are relevant to the Kit, then we should find those first before continuing with the Kit study.
• 6/2 @kelenelee removed P1 from this iteration of synthesis. Because there's only one dev in this study amongst the multiple designers and product managers, it's not clear what value synthesis of P1 will bring.
• 6/13 @kelenelee created a new issue specifically for interview transcription: Revise auto-transcriptions #434
• 6/14 P5 and P8 are de-prioritized from analysis because they were not officially their team's PM
• 7/18 @hdchangie presented a slide deck on designer participants
• 7/20 Transcripts were moved to google docs instead of spreadsheets to allow for coding via comments and highlights. P5 and P8 were coded
• 8/1 @kelenelee presented an initial slide deck summarizing just PM participants

ACTION ITEMS

Designers

• P10 @hannahcarlan
• P11 @LeslieCi
• P4 @hannahcarlan or @LeslieCi
• Meet to collaborate on synthesis of designer data
• Put together some kind of written summary

Product Managers

To preserve the old Step 2, re-analysis should be done in Step 4.

• P3 @kelenelee
• P6 @kelenelee
• P9 @kelenelee
• P2 @hdchangie
• P7 @hdchangie
• Meet to collaborate on synthesis of PM data
• Put together some kind of written summary

P8 @cunninghamej
P5

SUBTASKS

ALL ASSIGNEES PLEASE MAKE A COMMENT WITH THIS TEMPLATE

To encourage our team members in the habit of documenting their thought process, the template includes questions that you need to answer by editing the comment. I know that editing in Markdown can be a pain (sorry!) but I hope that the spaces will help the typing experience be less overwhelming. Let me know how my experiment goes for you.

- [ ] Read the **[updated plan](https://docs.google.com/document/d/1q9BdkerP6zwT0_TeVWqUwO6KI7YblVeU0OTORIJFxZ0)** 
- [ ] For learning about the **affinity diagramming technique**, you should watch **[this series of videos](https://www.youtube.com/playlist?list=PLPzC8RwtTOZkt7lukZU0pTZSw6D2Uok2H)** on a UXR study of home cooks. It is one of the only series where a senior UX person actually demonstrates UXR techniques. You can skip to the affinity diagramming video, but the other videos in the playlist can give you more context for the study. (The mini-series is part of the greater **[Micro-Lessons series](https://www.youtube.com/playlist?list=PLSaGB80mfUaSdlERUdhhQSbTBw214xWjS)** which includes resources on personas, surveys, coding, and more.)
- [ ] For any **[interviews recordings](https://drive.google.com/drive/folders/1G5uOImuMjhaWmc60p-wZ9c2PPLLbUXqV)** you rewatch, **clean up the transcript**. Add timestamps and remove typos and "um"s. If you have time and you're already there watching the video, this would help us quite a bit.
- [ ] For your assigned participants, **redo the affinity diagram** and use the **[updated plan's list of themes](https://docs.google.com/document/d/1q9BdkerP6zwT0_TeVWqUwO6KI7YblVeU0OTORIJFxZ0/edit#bookmark=id.ikcsa4d2na0o)** to guide you.
- [ ] Come up with **meaningful and helpful insights** that you can present to the rest of the research team
- [ ] Come back to your comment and **answer the questions** below
- [ ] Scroll back up to the main part of the issue and **check off your assigned tasks**.
___
### Any adjustments to the tasks you were originally assigned, or problems you ran into?



### Any significant discoveries, successes? New ideas for future research? 



### Any suggestions for affinity diagramming resources from experienced UX researchers? 



### Any other suggestions for improving your GitHub experience for this issue, or for GitHub overall?

[kelenelee]

• Read the updated plan
• For learning about the affinity diagramming technique, you should watch this series of videos on a UXR study of home cooks. It is one of the only series where a senior UX person actually demonstrates UXR techniques. You can skip to the affinity diagramming video, but the other videos in the playlist can give you more context for the study. (The mini-series is part of the greater Micro-Lessons series which includes resources on personas, surveys, coding, and more.)
• For any interviews recordings you rewatch, clean up the transcript. Add timestamps and remove typos and "um"s. If you have time and you're already there watching the video, this would help us quite a bit.
• For your assigned participants, redo the affinity diagram and use the updated plan's list of themes to guide you.
• Come up with meaningful and helpful insights that you can present to the rest of the research team
• Come back to your comment and answer the questions below
• Scroll back up to the main part of the issue and check off your assigned tasks.

---

Any adjustments to the tasks you were originally assigned, or problems you ran into?

• Not yet. I assigned myself quite a few participants since I conducted them already, but I might have bitten off more than I can chew, we will see.
• I hate how long it takes to edit transcripts... I wish there was a way to make that process faster.
• I had a different affinity diagramming video up but I've since then switched it out for a different tutorial.
• I also notice that as a new lead it's difficult to manage people and do individual contributor tasks. In the future, someone in a lead position should not be focused on doing contributor-level work

Any significant discoveries, successes? New ideas for future research?

Hopefully this workflow strategy of divide and conquer will help speed the analysis process up. Also, I wish we could have interviewed more than one developer, but given the flaws in the interview study, I'm not sure if that's worth it

Any suggestions for affinity diagramming resources from experienced UX researchers?

Follow Debbie Levitt and Larry Marine on LinkedIn!

Any other suggestions for improving your GitHub experience for this issue, or for GitHub overall?

I know the main issue is quite long. I hope it's readable for everyone. I tried to use # and ### headings to break up the text.

We used to have everybody make separate issues for any subtasks or repeated tasks. My intention of making things this way is in an effort for each issue/webpage to be a more contextual story. This issue is one of the first issues trying this newer system. Idk if my efforts will increase documentation and storytelling - we're kind of stuck in GitHub per org reqs.

[hdchangie]

• Read affinity diagramming tips
• Watch this video on affinity diagramming
• For any interviews recordings you rewatch, clean up the transcript. Add timestamps and remove typos and "um"s. This is optional but IF you have time and you're already there watching the video, this would help us quite a bit
• When you complete the analysis for the participants you're assigned to, scroll back up to the main part of the issue and check off your name.
• Come back to your comment and answer the questions below

Any adjustments to the tasks you were originally assigned, or problems you ran into?

Not sure what this question means.

Any significant discoveries, successes? New ideas for future research?

Some new findings:

Commondalities:

• persona: both P2 and P7 are career transitioners with different backgrounds before.
• P2 and P7 mentioned they would choose to stay longer in the team, which is partially derived from intrinsic reasons that they find value in working on the team, and partially from extrinsic reasons that they want to keep the team's knowledge and build a consistent team.
• both P2 and P7 are aware of the significance of having a shared practice and the team's memory, and both encountered troubles when onboarding people or had to re-do previous designers' works due to the lack of documentation and shared practices/knowledge. This insight can also derive from our explicit questions as prompts, rather than coming from an emerging trend from our data (lack of ethnographic observation data)
• shared pain points: onboarding relies heavily on individuals' memory and knowledge. usually, those who stay on the team the longest hold the memory.
• both P2 and P7 point out a very interesting tension between product speed/taking actions to move the product forward vs. documentation. It sounds like it's almost impossible to both move the product cycle forward and have good documentation practices. --> having a baseline of a DS in the organization should help with this.
• both P2 and P7 know very little about DS (however, P7 claims their design is very good already).

Findings unique to the two teams:

• P7 mentions that designers and developers have separate meetings since they don’t find it useful to know much about details--is it a misunderstanding? should designers and devs collaborate? can a DS bring designers and devs together in workflow?
• P2 believes that having a DS can be beneficial for onboarding experiences, having a base to work on and building a sense of belonging for volunteers. (again, this insight may only come from our explicit prompt instead of what P2 really thinks).

Any suggestions for affinity diagramming resources from experienced UX researchers?

sticky notes are super useful and this time I outlined a process to re-map each participant's own experience as volunteer, which I find helpful.

Any other suggestions for improving your GitHub experience for this issue, or for GitHub overall?

No, you've done a great job!

[hannahcarlan]

• Read affinity diagramming tips
• Watch this video on affinity diagramming
• For any interviews recordings you rewatch, clean up the transcript. Add timestamps and remove typos and "um"s. This is optional but IF you have time and you're already there watching the video, this would help us quite a bit
• When you complete the analysis for the participants you're assigned to, scroll back up to the main part of the issue and check off your name.
• Come back to your comment and answer the questions below

Any adjustments to the tasks you were originally assigned, or problems you ran into?

• I just spent about 2 hours trying to understand the analysis of Dev 1 and Designer 4. I can re-sort old stickies into new themes, but I'm not quite sure what the end goal of this exercise is. Are we trying to synthesize User needs? Make recommendations for next steps? Some clarification would be helpful so I know what the goal of the synthesis is.
• It was much easier to re-sort into themes while watching the interviews myself and reading through pre-existing stickies.

Any significant discoveries, successes? New ideas for future research?

• Different designers are developing their own DS in collaboration with developers
• There is some kind of HfLA component library that designers get access to (mentioned by Des#4)
• There is little-to-no existing framework for documenting work outside of Figma, where earlier versions of designs are preserved by some teams, but without meta-level explanation of decision-making processes
• Onboarding and guidance obviously a major issue. Designers relying upon external resources (YouTube, Google) to learn about design.

Any suggestions for affinity diagramming resources from experienced UX researchers?

Any other suggestions for improving your GitHub experience for this issue, or for GitHub overall?

[LeslieCi]

• Read affinity diagramming tips
• Watch this video on affinity diagramming
• For any interviews recordings you rewatch, clean up the transcript. Add timestamps and remove typos and "um"s. This is optional but IF you have time and you're already there watching the video, this would help us quite a bit
• When you complete the analysis for the participants you're assigned to, scroll back up to the main part of the issue and check off your name.
• Come back to your comment and answer the questions below

Any adjustments to the tasks you were originally assigned, or problems you ran into?

Any significant discoveries, successes? New ideas for future research?

Any suggestions for affinity diagramming resources from experienced UX researchers?

Any other suggestions for improving your GitHub experience for this issue, or for GitHub overall?

[cunninghamej]

• Read the updated plan
• For learning about the affinity diagramming technique, you should watch this series of videos on a UXR study of home cooks. It is one of the only series where a senior UX person actually demonstrates UXR techniques. You can skip to the affinity diagramming video, but the other videos in the playlist can give you more context for the study. (The mini-series is part of the greater Micro-Lessons series which includes resources on personas, surveys, coding, and more.)
• For any interviews recordings you rewatch, clean up the transcript. Add timestamps and remove typos and "um"s. If you have time and you're already there watching the video, this would help us quite a bit.
• For your assigned participants, redo the affinity diagram and use the updated plan's list of themes to guide you.
• Come up with meaningful and helpful insights that you can present to the rest of the research team
• Come back to your comment and answer the questions below
• Scroll back up to the main part of the issue and check off your assigned tasks.

---

Any adjustments to the tasks you were originally assigned, or problems you ran into?

Any significant discoveries, successes? New ideas for future research?

Any suggestions for affinity diagramming resources from experienced UX researchers?

Any other suggestions for improving your GitHub experience for this issue, or for GitHub overall?

[kelenelee]

At the 6/6 all-hands meeting I presented what we've found so far with the PM analysis.

• So far no personas yet - Still working on P3 and P8 - consistent characteristics across participants so far
• Participants are junior-level career transitioners coming to HfLA to learn new skills
• Low product management skill level, even in participants with prior experience in the workplace and in product development. This leads to suboptimal product development workflows.
  • Documentation and action are competing for volunteer’s time
  • Teams have no UXR or market research to determine product-market fit
• Design and dev teams don’t collaborate because the team is not at a building stage yet

As for this study's methodology,

• We don’t have the expertise to evaluate PM skill set
• Contextual inquiry would have been a more effective method than self-reported interview data

Notes on what has been found about the designers so far

• No emphasis on documentation - move fast and get the work done
• Mismatch between stakeholder wants and what people want
• Lack of mentorship from seniors - more peer “mentoring”
• People’s goals for joining HfLA - how to connect that to documentation? How to INCENTIVIZE newbies?

Feedback from Yas

• Specify “lack of documentation.” Improve the documentation - how ??? it’s not a linear process at all - things need to be done and redone - WHAT should be documented? How much? Documenting everything would be difficult / impossible
• What is documentation vs proper workflow?
• Can interviews from one group solve something from the other group?

[kelenelee]

6/7 research meeting was cancelled.
6/14 research meeting, most of it was a group session of PM synthesis with Eric, Dongchen and me in the affinity diagram. Leslie and Hannah have already finished their affinity diagrams for the designer participants and are now working on a list of insights in this doc.

Haven't finished PM synthesis yet, so we're meeting later this week to wrap it up. Creating a set template for affinity diagram synthesis has been hard for me – I never feel like it's done, or it's the best way to organize our data. This has led to some confusion for Eric and Dongchen when they were creating stickies on their own.

We plan to create a similar doc for PMs like Leslie and Hannah's later.

Also, analysis takes a loooong time. I have spent hours each day on this, on #434... overall I'm getting the sense that this 6-10 hours a week is a ridiculous minimum given business expectations for output pace. We need some stopwatch to actually time how long work takes for people...

[kelenelee]

6/16 @hdchangie and I met up to for another working session. We decided that P8 should be left out of the synthesis for now, along with P5, because those participants had PM responsibilities but were not PMs in title. We are already moving pretty slowly and we will add P5 and P8 back if there's time in early July.

After wrapping up the stickies transfer that we didn't finish on 6/14, we started looking at individual sections of stickies to look for patterns. We connected stickies with lines. Some stickies were still missing at this point and we had to add them in to finish the pattern.

For each section of stickies, we'll write a summary in this doc.

[kelenelee]

I spent 3 hours today drafting the PM synthesis doc.

[kelenelee]

this is basically done. designer synthesis is here. see #402 for pm synthesis