Sound Design Consideration: Investigate Stereo & Panning options/possibilities

[terracoda]

Based on interview feedback, and will post related feedback if I get more over the next two weeks.

Feb-P1 participant had a good exploration of Molecules and Light and suggested that stereo as an option and panning sounds could improve the experience when people are using headphones.

Panning was suggested for indicating the movement of photons from left to right.

Provide a stereo mode setting - adding more stereo components - the pitches of the light could be panned. That would make it clear that you are shooting a photon at a molecule. But it works.

Sound design team could consider investigation of stereo and panning features.

[Ashton-Morris]

I have thought about this idea before and I feel stereo imagery could bring a lot of information to the sims sound.

Thinking @jbphet could probably put the sound source on the actual image of the photon and have it pan using that location information. Will be good to talk about tomorrow.

Also it would be good to get Wanda's input on this tomorrow

[jbphet]

I implemented a prototype of panning for the photon emission sound and tried to demo it over zoom, but people didn't seem to be able to hear it. I'll work on implementing support for panning to the point where it can be checked in, and then will publish a dev version so that people can try it out on their own machines.

[Ashton-Morris]

I added a few files here

• vibration-spacialized.mp3 (this goes from smallest width-mono to largest width-full stereo with the vibration) it is hard to perceive though
• rotate-directions-002-spacialized.mp3 (these go through a rotary speaker cabinet effect that make it sound like its going around. The rotation speed is not exact though because the speaker did not have those settings, so if we like this direction I can recreated the effect from scratch with specific numbers)
• rotate-directions-001-spacialized.mp3 (these go through a rotary speaker cabinet effect that make it sound like its going around. The rotation speed is not exact though because the speaker did not have those settings, so if we like this direction I can recreated the effect from scratch with specific numbers)
• photon-000-spacialized.mp3 (These pan from left to right with the speed of the photon)
• photon-003-spacialized.mp3
• photon-002-spacialized.mp3
• photon-001-spacialized.mp3

[jbphet]

I integrated the new spatialized sounds for photon emission (initial emission only), rotation, and vibration, and published these changes in a dev version (see https://phet-dev.colorado.edu/html/molecules-and-light/1.5.0-dev.34/phet/molecules-and-light_en_phet.html). These changes were reviewed with the sound design team during the 3/3/2020 MAL sound design meeting, and the general conclusion was that the effect is pretty subtle and, in some cases, possibly not an improvement. For example, @emily-phet felt that having the sound for initial photon emission being so much in the left ear make it seem a bit "disembodied from the sim".

Our plan is to try some variations on the photon emission sounds to see if we can come up with something that we feel enhances the user experience and then, based on the outcome of that exercise, decide whether we want to make use of spatialization as a design dimension for this sim and at this time.

@Ashton-Morris - can you please provide some variations on the photon emission sounds that can then be integrated into the MAL sim with an options dialog for selection? I'm thinking something like the following (though feel free to use your judgement and creativity to come up with others or alternatives if you like):

• a sound that starts just a little left of center and that pans from left to right very rapidly
• a sound that starts all the way left and rapidly pans all the way right so that we can hear the full effect
• a sounds that starts a little left and moves about the rate of the visual motion, so is as close as possible to what would be heard from something that was traveling and emitting a sound
• a sound that is at about the correct amount of left-ness for the initial emission and doesn't pan at all (for this one I'm thinking, hey, it's not like a bullet makes sound as it moves, it just makes sound when it's fired)

[Ashton-Morris]

Ok here is what I have.

• photon-stays-little-left-003.mp3
• photon-stays-little-left-002.mp3
• photon-stays-little-left-001.mp3
• photon-stays-little-left-000.mp3
• photon-rate-of-visual-motion-003.mp3
• photon-rate-of-visual-motion-002.mp3
• photon-rate-of-visual-motion-001.mp3
• photon-rate-of-visual-motion-000.mp3
• photon-all-the-way-left-to-right-pans-rapidly-003.mp3
• photon-all-the-way-left-to-right-pans-rapidly-002.mp3
• photon-all-the-way-left-to-right-pans-rapidly-001.mp3
• photon-all-the-way-left-to-right-pans-rapidly-000.mp3
• photon-little-left-pans-rapidly-003.mp3
• photon-little-left-pans-rapidly-002.mp3
• photon-little-left-pans-rapidly-001.mp3
• photon-little-left-pans-rapidly-000.mp3

[emily-phet]

Everyone,
Please take a listen to these as well and weigh in on the following (below). Please use the checkbox list below to let us track who has taken a listen and added all their comments.

• @terracoda
• @Matthew-Moore240
• @Ashton-Morris
• @jbphet

Here are the questions on the table:

1. Is there a more refined version of the stereo sound design that we prefer over the current (v34) one.
2. Given the best design options we have currently for MAL, what are the costs/benefits of having stereo in the sim at all?

Some possible "costs" to consider include: inclusion of stereo increasing design complexity, potential implementation complexity, QA complexity, and possibly other downstream impacts - such as teacher resources, etc.

[Matthew-Moore240]

I really like the stereo sound. Some of the interviewees have mentioned that stereo sound would enhance there experience, so I think if its possible to add it even for just the rotating molecules would be good as long as it wasn't jarring. The dream of course is to have an option for a user to pick either mono or stereo sound and kind of tailor there own experience, but I think John talked briefly about that being difficult.

[jbphet]

In the 3/10/2020 sound design meeting we decided to try stereo sounds for rotation and vibration but leave the photon sounds as mono for now.

@Ashton-Morris - To do this, I'll need stereo versions of the slow motion versions of these sounds to meet this goal. I'll assign this to you, please assign back to me once we have them.

[jbphet]

I listened to the sounds listed in #284 (comment), and my favorite was photon-rate-of-visual-motion-xxx. The one "stays-little-left" versions weren't bad, but the motion of the other seemed good to me. The others were generally a bit too much.

If we end up using a stereo sound, I think the "rate-of-visual-motion" ones would be best, but in my opinion they shouldn't pan quite so far to the right.

[emily-phet]

At last meeting - EM suggested that we consider the panning for the photons separately from spatialization for other sounds.

From the March 10th meeting, seemed all were on board with the stereo versions of the rotation and vibration sounds, we didn't get to discussing decisions related to panning for the photon sound.

Let's go ahead and move forward with the spatialization for the rotation and vibration sounds. Let's discuss spatialization of photon sound March 17th.

[emily-phet]

@Ashton-Morris - just noticed a note from @jbphet here that they'll need to be spatialized versions of the sound in slow motion. Can you create these for Tuesday as well?

[Ashton-Morris]

Here you go

rotate-directions-002-spacialized-slow-mo-longer.mp3
rotate-directions-002-spacialized-slow-mo-longer.wav
rotate-directions-001-spacialized-slow-mo-longer.wav
rotate-directions-001-spacialized-slow-mo-longer.mp3
vibration-spacialized-slow-mo-loop.wav

[jbphet]

I’m also noticing it on the spatialized vibration loop, so can I get a longer version of that one too? Thanks, John From: Ashton-Morris [mailto:[email protected]] Sent: Tuesday, March 17, 2020 9:27 AM To: phetsims/molecules-and-light <[email protected]> Cc: John Peter Blanco <[email protected]>; Assign <[email protected]> Subject: Re: [phetsims/molecules-and-light] Sound Design Consideration: Investigate Stereo & Panning options/possibilities (#284) Here you go<https://drive.google.com/open?id=1UdeAk6MxXEZMSj2Nt6tDi35R-jej5ppM> rotate-directions-002-spacialized-slow-mo-longer.mp3 rotate-directions-002-spacialized-slow-mo-longer.wav rotate-directions-001-spacialized-slow-mo-longer.wav rotate-directions-001-spacialized-slow-mo-longer.mp3 vibration-spacialized-slow-mo-loop.wav — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#284 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAXQUT6LUHJHKZODBM5UYMLRH6JGDANCNFSM4KX4VXKA>.

[Ashton-Morris]

vibration-spacialized-slow-mo-longer.wav
vibration-spacialized-slow-mo-longer.mp3

[jbphet]

I need the vibration, spatialized, normal speed loop as well.

[jbphet]

I have build a dev version with the latest spatialized sounds for vibration and rotation (including slow motion), and there is a query parameter called useStereoSounds to turn on/off the spatialized sounds. However, there seems to be some sort of "scratchiness" that is inherent in the vibration sound, so it's a bit difficult to compare. We should review and see if we can "hear beyond" the issues to decide if we want to go with stereo or mono sounds for this.

Link with mono sounds: https://phet-dev.colorado.edu/html/molecules-and-light/1.5.0-dev.38/phet/molecules-and-light_en_phet.html

Link with stereo sounds: https://phet-dev.colorado.edu/html/molecules-and-light/1.5.0-dev.38/phet/molecules-and-light_en_phet.html?useStereoSounds

[Matthew-Moore240]

Notes for the rotation sound effect

Slow Motion

1. Counter clockwise doesn't sound stereo and sounds more like a continuous fog horn
2. Clockwise in slow motion sounds good

Normal Speed

1. Counter clockwise sounds good
2. Clockwise sounds quiet and seems to get cut off near the end

[jbphet]

In the 3/17/2020 sound design meeting we decided to use mono sounds for the photons and continue to develop the stereo sounds for vibration and rotation.

[Ashton-Morris]

I hope I got the right rotation one. This one sounded less like a foghorn to me. Files Here

• vibrate-longer-spatialized-slow-mo.mp3
• vibrate-longer-spatialized-slow-mo.wav
• vibrate-longer-spatialized.mp3
• vibrate-longer-spatialized.wav
• rotate-directions-001-reverse-spatialized.mp3
• rotate-directions-001-reverse-spatialized.wav
• rotate-directions-001-reverse-spatialized-slow-mo.mp3
• rotate-directions-001-reverse-spatialized-slow-mo.wav

[jbphet]

Note to self: Once these sounds are finalized, @Ashton-Morris and I should revise their names in GitHub to something clear and concise, and should store any asset files that would help him to recreate and/or revise these in the future.

[Matthew-Moore240]

Panning for the rotation notes.

1. If rotating clockwise I think starting hard in the left ear and moving right gives the rotation more movement to clockwise. The same thing for counter clockwise.

[jbphet]

We discussed @Matthew-Moore240's comment immediately above during the sound design meeting on Mar 24 and decided that reversing the sounds may be sufficient to address the concern, so I'll make that change.

[jbphet]

Reviewed during the 3/31/2020 issue and we decided that the mods to the rotation sounds (swapping CW and CCW) addressed the issue, so we should be good to go with those sounds.

[terracoda]

@jbphet, I've been testing the descriptions (with sound) in very recent versions. The sounds are sounding pretty good to me, so I am un-assigning for now. Please re-assign if you need anything from me, or close if this issue is now stale.

[jbphet]

I think it's safe to close this one. The sound design is getting a final round of review in #233, so if anything comes up in that issue that relates to this one, we can reopen, but right now things are looking (and sounding) good.