Implement sound design

[jbphet]

@Ashton-Morris, @emily-phet, and I (@jbphet) did some brainstorming on the sound design for the proportion sim and thought we should write some of these things down.

• Use two sound generators that are producing the same note when in correct proportion, and when the proportions aren't correct they would produce different notes and thus beat frequencies.
• Same idea - two sound generators - but produce the notes periodically so that it doesn't drive people crazy with continuous sounds
• Two rhythmic beats that are distinct and line up when the proportions are correct - in a fun way, using double time, triplets, etc, but get out of sync if the proportions are incorrect. They'd speed up or slow down the overall tempo as the fundamental sizes change.

[jbphet]

I made a quick recording to demonstrate beat frequencies. In the explanation at the beginning I probably should have said that I'm "playing the same note on two different strings" instead of that I'm "playing the same note".

Here's a link: https://drive.google.com/file/d/1AmrwecgS4_QDzztTGWdeK7pM2TMpIomE/view

[zepumph]

In the above commit. I made a refactor that exposed what I call the proportionFItnessProperty in ProportionModel. This is a value between 0 and 1. When 1 the proportion is "perfect", and 0 means it is outside of the tolerance range. I did this with the hope that it would be easy to monitor for a simple "fitness" sound that can change in the same way as we do for color. Let me know if I can be of more help.

In my opinion having something (even very simple) implemented by next meeting seems very valuable for discussion. Good luck!

[emily-phet]

@Ashton-Morris - please propose something (could be related to beat frequencies or anything else) that is reasonably pleasant and can be added to the sim quickly. I'd like to get some sound in asap so there's some auditory feedback available as the team brainstorms sound and other features. Let's start with something that aligns with the color change in the sim, as @zepumph indicated.

@jbphet @zepumph Please work together to add the first sound. @jbphet if you're able to begin showing @zepumph the ropes on how to add sound to sims with Tambo, that would be ideal, but if depending on where you're at with your other rocks, you might do this later in the quarter and proceed with whatever is fastest now.

[Ashton-Morris]

@emily-phet I have a few ideas ready but I cant seem to properly clone the directory that has Proportion.

It would be much quicker if someone could do a simple screen recording of a proportion slider slowly going up and down. I can then do a few mockups of how sound would work and can upload it here.

If anyone is willing to do that please let me know.

[emily-phet]

@Ashton-Morris - I'll try to do a screen recording for you later today. Here's a link to the latest dev version, in case that's helpful in the meantime.
https://bayes.colorado.edu/dev/html/proportion/1.0.0-dev.5/phet/proportion_en_phet.html

[Ashton-Morris]

@emily-phet Don't worry about a screen recording, I can do that. I didn't know there was a dev version. That all I needed.

I'll work on some mockups asap

[emily-phet]

@Ashton-Morris I figured that might have been the issue. :)

When you make the screen recording, keep in mind that in some scenarios, learners will be moving each object one at a time, but in the more desirable scenarios, they're moving both at the same time (e.g., on a touch screen, or with tangible objects). Just thought that might impact your thoughts re sound.

[Ashton-Morris]

I worked on some ideas this morning. You should be able to view these in your browser. I will do my best to explain my thinking with each one. @emily-phet Please assign the proper people for review

• Proportion_C_Major_Sine.m4v
  Max out of proportion = Low volume and only a C sine wave tone
  Max in proportion = Added E and G notes that fade in which creates a C Major Chord
  What you're hearing = A C Major Chord fading in and out along with proportion

• Proportion_Fifth_Sine.m4v
  Max out of proportion = Two C Sine Waves and No Volume
  Max in proportion = Two Sine waves C and G creating a perfect fifth
  What you're hearing = A kind of wobble/dissonance as they are out of proportion and a cleaning up/harmony sound when they are in.

• Proportion_One_Semitone_Sine.m4v
  Similar to above
  Max out of proportion = Two Sine Waves one C and one B and No Volume
  Max in proportion = Two Sine waves playing the same note C
  What you're hearing = A kind of wobble/dissonance as they are out of proportion and a cleaning up/unity of the notes sound when they are in.

• Proportion-Rhythm.m4v
  Max out of proportion = A single rhythm at a low volume
  Max in proportion = That same rhythm being played at 2X and 4X the speed fading in in volume
  What you're hearing = A kick drum like sound going by it self then a more complex rhythm rising in volume as Proportion is reached.

• Proportion-Vibrato.m4v
  Max out of proportion = A single tone with vibrato and a low volume
  Max in proportion = A single tone with no vibrato
  What you're hearing = A single tone going from wobbly/vibrato to straight and pure and louder as Proportion is reached.

• Proportion-Random-Clicks.m4v
  Max out of proportion = A single repeating thump/click that has it's pitch randomly changed by 16 semitones each time it repeats at a low volume
  Max in proportion = A single repeating thump/click playing a consistent rhythm at a louder volume
  What you're hearing = A single repeating thump/click that goes from being in disarray (playing random versions of itself) then fading into a consistent rhythm when the sim is in Proportion.

[emily-phet]

@Ashton-Morris Thanks for these!
@zepumph @jbphet Can we have a version with two sound options - with "Proportion-Vibrato" and "Proportion-Random Clicks" (random clicks as the default)?
If it's not time consuming to do, having a third option of "Proportion_C_Major_Sine" would be nice to have in as well.

To ensure those playing with at-home tangible versions has some auditory feedback to consider, this would be my top priority effort for Proportion development this week.

[emily-phet]

@Ashton-Morris Here are my thoughts:

• C_Minor_Sine - this sounds very pleasant to my ear. My concern is that from the video I wasn't sure if the feedback tied strongly enough to the goal state of "at ratio". That state needs to simultaneously stand out clearly. If this one gets implemented in the first options set, my hunch is that it will feel like it stands out more so than watching the video. Let's see!

• Fifth Sine and One_Semitone_Sine - both of these sounded slightly less pleasant to my ear than C_Minor_Sine, and had the same potential issue of not quite strong enough of a highlight of the "at ratio" state.

• Rhythm - I thought this approach might muddy the waters a bit at cueing the 'at ratio' state. If we feel like pursuing a rhythm-based approach makes sense to move forward with, I think we'll want one with a really strong change when very very close, and right at, ratio.

• Vibrato - I thought this one had the strongest cueing to "tune" the ratio, though slightly less pleasant to my ear than the C_Minor_Sine, and the random clicks.

• Random Clicks - I really liked this one, and I'm curious what others think. It sounds interesting and pleasant, while also providing some strong cueing when you're "at ratio".

[Ashton-Morris]

I have a single click and a loop of them below for implementation.

In my DAW I have the clicks set to randomize between 0 and +16 semitones. The Randomizer's rate (like the rate of an LFO) is set to 30HZ. Then its set to 100% out of proportion = 100% of that randomizer effecting the clicks, and In proportion = 0% that randomizer effecting those clicks.

proportion-random-clicks-loop.mp3
proportion-random-clicks-single.mp3

For the Proportion_C_Major_Sine you could play sine's with web audio. That would be: Play C3 when it is 100% our of proportion and then attach the volume of the sine waves E3 and G3 to how in proportion the sim is. Thats how I had it set up.

Edit: These things can be tricky with web audio, we can talk about implementation hurdles later today ^__^

[zepumph]

I, with tons of great help from @jbphet, added modes for the vibrato and random clicks sounds. The random clicks is on by default. The vibrato is currently just two sine waves at 220hz where one can be moved up to 230hz depending proportion fitness.

[zepumph]

After the above commits, we successfully have mechamarkers input working with the two sounds discussed during today's meeting.

Summary of what is currently in the sim:

• A prototype of Vibrato
• A prototype of Random Clicks
• A prototype that just changes pitch based on the fitness
• Sound working for keyboard/drag/marker inputs

Next I will try to add C_Major sine as the final step for the day.

[emily-phet]

@zepumph There were times when I thought I was hearing sound feedback when the screen looked white. Is this the case, or is the tolerance such that the screen just happens to look white because it's a super subtle shade of green?
If it's the case that the background is exactly white but providing sound feedback, we should discuss if we want this to happen tomorrow.

[zepumph]

After 3523bc9, now the ?movingInProportionThreshold query parameter takes the value in "distance from max fitness" instead of distance from min fitness. So the value used to be .7, but now it is .3.

[zepumph]

Notes from adding the new combined wav sound:

• I added the new wav file from @Ashton-Morris and named it moving-in-proportion-loop-peaceful-and-choir.wav.
• I can't actually hear a difference between the two sounds I combined in code, and @Ashton-Morris's. Did you take out some of the bass notes as described in Implement sound design #9 (comment)? I think my ears may be failing me.

[zepumph]

Everything has been implemented from the above checkboxes.

If it's the case that the background is exactly white but providing sound feedback, we should discuss if we want this to happen tomorrow.

I have not seen this, but will be on the lookout. Please create a new issue if you can consistently reproduce. I'm pretty sure that it is just a very faint shade of green. Hopefully logging added in #103 will help see that.

---

My thoughts about combos:

• I agree with @emily-phet that option 2 is a bit better than option 1, and I like fifths more than chords (yay fifths 2)!
• I like the choir+ sound most.
• Thus my vote is fifths2 + Choir+ peace
• I think that fifths 2 could use some minor tweaking to have a bit less twang at the end. It is interesting on its own, but it is a bit muddy when combined with the "moving in proportion" sound. I feel this may no matter what moving in proportion sound is selected.

Looking forward to tomorrows chat!

[jbphet]

As requested by @emily-phet, I spent some time reviewing the latest dev version (https://phet-dev.colorado.edu/html/ratio-and-proportion/1.0.0-dev.41/phet/ratio-and-proportion_en_phet.html) and have the following comments:

• The hysteresis for playing the "in proportion" sound is too much when not doing dual drag. I know I advocated for having some hysteresis, so I should elaborate. When using the sim on a touchscreen and moving both markers, the "in proportion" sound played and stopped a lot, and this seemed distracting, so hysteresis seemed important. However, when using v1.0.0-dev.41 with a mouse, it feels odd that the sound isn't played each time the optimal proportion is reached and the max green is shown. So, perhaps the hysteresis should only be used when both markers are being dragged, or maybe the amount should be much less when one marker is dragging versus both markers. This problem was particularly noticeable when using fine-grained keyboard-based movement (shift-arrow).
• The "in proportion" sound should play every time the threshold is crossed. As it is now, some fast crossings don't cause it to play.
• My favorite combination of sounds is also "Choir/peaceful tranquility blend" and "Fifths option2". The former seems like the clear winner to me for the "Velocity Sound". For the latter (currently labeled "Staccato Success Sound"), I would also be okay with "Arpeggiated maj7 chord" or "Single ding high note".

[zepumph]

The hysteresis for playing the "in proportion" sound is too much when not doing dual drag.
and
The "in proportion" sound should play every time the threshold is crossed. As it is now, some fast crossings don't cause it to play.

This was fixed in master in b643b7a and was a bullet from a previous design meeting, see
#9 (comment):

The hysteresis option is weird when just moving a single value, as fine tuning it can yield just a flash of green, and no sound. Perhaps it should only be implemented for when moving in proportion.

[brettfiedler]

The velocity sounds are reaaalllyyy quiet for me relative to the in/out of prop sounds, so this was a little harder to assess how rich/crowded the soundspace was, but:

• MovingInProp: Choir/Peaceful Choir are on nearly equal footing for me. I can see how the soundspace could be a little overpacked. I'd be fine with either. I suspect those using external speakers won't really hear the added parts in PeacefulChoir.
• InProp: I am fine with either Option 2 or the Arp Maj 7th. I might like the Arp Maj 7th the best if the notes the choir was singing meshed up a bit better? Something about the softness of that success sound is still appealing to me for repeated listens.

Could we maybe consider playing a minimum amount of a success sound that we consider the "core" part before it rings out? It does feel a little weird to let go and have some of the arpeggiated sound cut out so quickly. This may also address John's comment on sounds sometimes not playing. I suspect it may be triggering, but getting cut off just as quickly?

[zepumph]

From sound design meeting today:

• Moving in proportion sound is much quieter than the in proportion sound. We should balance those by probably meeting somewhere in the middle.

• We decided as a group on Choir + Peaceful and Fifths 2. It may need some fine tuning, but we want to go through a round of interviews before doing more work on this (in case we need to pivot).

• Clean up all other sound options

• (not needed for interviews) When you would hear the staccato sound, don't play pick up and drop UI sounds for the hands.

[zepumph]

Some notes to look at when (if) we come back to fine tune the Choir + Peaceful and Fifths 2 sound combo:

• For now we are keeping the bassy breadth of the peacful tranquility sound in because it sounds "complete," but in the future, we may change how we feel to thinking it a bit too muddy. We will come back to this thought
• The balance of levels between Moving in proportion sound vs in proportion will likely need to be tweaked beyond what is done for Implement sound design #9 (comment).
• We may want to make tweaks to the fifth option 2 sound to blend better with the nirvana sound.

[zepumph]

Once the above is implemented, we can close this issue. As stuff comes out of interviews, we can make more issues for it.

[zepumph]

Alright! All has been implemented and cleaned up. Let's create new issues from here as we fine tune our sounds. Good work team!