Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explore new sound design focused on all quadrilateral shape transformations #175

Closed
brettfiedler opened this issue Jul 7, 2022 · 51 comments

Comments

@brettfiedler
Copy link
Member

brettfiedler commented Jul 7, 2022

We aim to investigate a new sound design that emphasizes the transition between all named quadrilateral shapes, while supporting their maintenance as well. We aim to capture the build-up of unique properties for named shapes, but not contain the design space so tightly to require exact mapping of properties to sounds. The conceptual space is very complex and does not lend itself well to simple sound design for all relevant conceptual properties.

@brettfiedler brettfiedler changed the title Shape Property Reference Sounds for all quadrilaterals Explore new sound design focused on all quadrilateral shape transformations Jul 19, 2022
@brettfiedler
Copy link
Member Author

Talked with EM and Ashton about a possible general sound design that emphasizes the transitions between shapes we name (Convex Quadrilateral, Concave Quadrilateral, Dart, Kite, Trapezoid, Isosceles Trapezoid, Parallelogram, Rhombus, Rectangle, Square) and give the overall impression of building up as you move from shapes with no discernible properties (convex quadrilateral, only requirement = interior angles sum to 360) to shapes with rigid definitions (most complex being square which inherits the definitions of most other quadrilateral shapes).

I attempted to bundle together the unique properties (angles, lengths) about each named quadrilateral to indicate the smallest possible changes as you change between shapes. This resulted in about 7-8 categories as pictures below.

image

  1. Quadrilateral (ambient or silent)
  2. Concave (or inverted)
  3. Diagonal symmetry
  4. Lateral Symmetry
  5. Half-Parallel (1 set of parallel sides)
  6. Full-Parallel (2 sets of parallel sides)
  7. Angle Equality (all = 90)
  8. Side equality (all sides same length)

Categories such as Convex/Concave or Half-Parallel/Full-Parallel can replace each other. Otherwise the sounds are additive.

Likely constraints for any sound designs:

  • Only play new sounds on a transition (with some extra attack on first occurrence)
  • When resuming drag that does not change the shape, fade in sounds (no attack)
  • (idea) Scale volume with approximate area of shape (bounded to reasonable volumes)
  • Should add some quality to the sounds that improve their listenability when maintaining the same shape.
  • Sounds do not need to map exactly to properties in a quickly decipherable way - acknowledgement of transition and pleasantness of sounds should be priority.

@brettfiedler
Copy link
Member Author

brettfiedler commented Jul 19, 2022

At sound meeting today we talked about introducing further musicality into the sound design during this rapid exploration, including the possible introduction/disappearance of instrumental/rhythmic components.

@Ashton-Morris will begin some mock-ups and follow up with @BLFiedler for synchronous design time and otherwise ping design team members for feedback here/in Slack.

@brettfiedler
Copy link
Member Author

brettfiedler commented Jul 25, 2022

Made by @Ashton-Morris this week:

At a quick meeting between AM and me, we discussed showcasing the possibilities of the following tomorrow using his software:

  • Listening with more randomization in what aspects are present
  • FMOD (music/rhythm layering example above):
    • Different sets of sounds present (e.g., 1246, 15678)
    • Emphasize one instrument at a time (louder) but play entire track at all times
    • Play a basic rhythm and add one later per shape (12, 13, 14, 18, etc)
    • Play basic rhythm (and maybe 1 other track) and alter instrument voice for each shape
    • Minor mapping of sounds and mixing beats with ambience/tones
      • E.g., We know convex or concave will be the base layer for all shapes, so keep them pretty inconspicuous as a simpler (warbling) tone and start adding more interesting beats with each shape (or different)
  • Adding different parts to the beat (completing the track) vs. layer additional, complementary tracks
  • Individual Notes (like streamable link above)
  • Different ambiences per shape (like streamable link above)

We'll hold off exporting sounds until after the meeting, but have them ready for when @jessegreenberg can implement.

@brettfiedler
Copy link
Member Author

@Ashton-Morris will export the tracks/loops/tones he has created so far broken out by the sets he demo'd at the last sound meeting. Then @BLFiedler will give specific sound designs for @jessegreenberg to implement in the sim as a priority over other issues (except Voicing issues).

I made these notes on the sets that were demo'd.

Individual notes (chord building for each aspect of shape, e.g., (from https://streamable.com/lqd5qp))
Ambiences (each shape one, ambience may reflect aspect of shape, e.g., https://streamable.com/9q0a07)
Tracks: Beat tracks layer with each category addition (building) (e.g., https://streamable.com/mx8i1m)
Tracks: Base beat + new track per shape (new track may reflect aspect of shape)
Arpeggio: New part of arpeggio per shape (building up)
Arpeggio: New part of arpeggio (building up) + new beat track

Tagging other Quad sound design folk for awareness - @emily-phet @terracoda

@emily-phet
Copy link

Just a note @BLFiedler: If you feel some of these seem more promising than others, you can go ahead and pick those that seem most promising to move forward with. It's definitely hard to predict exactly how they'll feel before you get to play with them in the sim...but wanted to make sure you knew it's ok to go with your gut to help streamline the process. If your gut says you need to see them all in action, of course, go for it. :)

@Ashton-Morris If any of these don't make it into the sim, can you record a short demo of the idea and list in this thread, so we have a record of it? Since we're trying something totally new here for Quad, we may end up writing about it later and having some examples of early designs will help!

@brettfiedler brettfiedler removed their assignment Jul 27, 2022
@Ashton-Morris
Copy link

In my option we should start with three (time permitting).

  • (3.) Arpeggio: New part of arpeggio
  • (5.) Melody: Base track plays consistently and each shape has a corresponding melody to it
  • (4.) Tracks: Beat tracks layer with each category addition (building)

@jessegreenberg There should be two types of logic from what I can tell, Shape Identifier, and State Identifier. The sound mockups wen't off of both directions. Hopefully the audio tracks could be interchangeable once that logic is put in. (ex. the logic for the Ambience mockup is the same for the Melody mockup, and the logic for building music is the same for the Arpeggio one.)

Let me know if you have any questions regarding implementation, I'd be happy to walk though things if needed.

Also, I'll do a screen recording of each mockup for our records.

Here are all the files to test.

  1. Individual notes (these are actually lowest in pitch from highest in number to lowest)
    Quad_Individual_Notes-000.mp3
    Quad_Individual_Notes-007.mp3
    Quad_Individual_Notes-006.mp3
    Quad_Individual_Notes-005.mp3
    Quad_Individual_Notes-004.mp3
    Quad_Individual_Notes-003.mp3
    Quad_Individual_Notes-002.mp3
    Quad_Individual_Notes-001.mp3

  2. Ambiences
    Quad_Ambiences-005.mp3
    Quad_Ambiences-001.mp3
    Quad_Ambiences-003.mp3
    Quad_Ambiences-002.mp3
    Quad_Ambiences-004.mp3
    Quad_Ambiences-007.mp3
    Quad_Ambiences-000.mp3
    Quad_Ambiences-006.mp3

  3. Arpeggio: New part of arpeggio
    Quad_Building_Up_Arpeggio-000.mp3
    Quad_Building_Up_Arpeggio-001.mp3
    Quad_Building_Up_Arpeggio-002.mp3
    Quad_Building_Up_Arpeggio-003.mp3
    Quad_Building_Up_Arpeggio-004.mp3
    Quad_Building_Up_Arpeggio-005.mp3
    Quad_Building_Up_Arpeggio-006.mp3
    Quad_Building_Up_Arpeggio-007.mp3

  • Optional Shape Identifier Tracks for above
    Quad_Building_Up_Arpeggio_Optional_Shape_Identifier_Tracks-000.mp3
    Quad_Building_Up_Arpeggio_Optional_Shape_Identifier_Tracks-001.mp3
    Quad_Building_Up_Arpeggio_Optional_Shape_Identifier_Tracks-002.mp3
    Quad_Building_Up_Arpeggio_Optional_Shape_Identifier_Tracks-003.mp3
  1. Tracks: Beat tracks layer with each category addition (building)
    Quad_Beat_Tracks_Building_Base_Rhythm.mp3
    Quad_Beat_Tracks_Building_Building_Tracks-000.mp3
    Quad_Beat_Tracks_Building_Building_Tracks-001.mp3
    Quad_Beat_Tracks_Building_Building_Tracks-002.mp3
    Quad_Beat_Tracks_Building_Building_Tracks-003.mp3
    Quad_Beat_Tracks_Building_Building_Tracks-004.mp3
    Quad_Beat_Tracks_Building_Building_Tracks-005.mp3
    Quad_Beat_Tracks_Building_Building_Tracks-006.mp3

  2. Melody: Base track plays consistently and each shape has a corresponding melody to it
    Quad_Melody_Tracks_Base_Music.mp3
    Quad_Melody_Tracks_000.mp3
    Quad_Melody_Tracks-005.mp3
    Quad_Melody_Tracks-004.mp3
    Quad_Melody_Tracks-002.mp3
    Quad_Melody_Tracks-003.mp3
    Quad_Melody_Tracks-001.mp3

@jessegreenberg
Copy link
Contributor

Thanks @Ashton-Morris!

Sorry, I read through the issue but dont understand what to implement. @Ashton-Morris or @BLFiedler can you describe how each of the sound design should work? What sound is associated with what shape or state? How long should each sound play in response to input?

@brettfiedler
Copy link
Member Author

brettfiedler commented Aug 1, 2022

For some quick initial testing let's go with 4 sound sets.

  • Let's use a general rule for persistence for all of them for now (though we may choose to modify these for any individual set ahead of interviewing): Sounds play for 2 seconds after (mouse/touch release, last keyboard input) of any vertex or side

1. Tracks - Build Up.

Here we should try assigning a track to a specific element of the shape that may vary between shapes (e.g., amount of parallel), but builds toward more complex when you reach a square. This is based on my initial comment at the top of the issue.

Ideally, tracks that are already playing continue to play without restarting their loop. Functionally this might involve having all tracks playing all the time, but their volumes muted until they reach a shape that has that track assigned?

First, we assign each file a letter for brevity in the shape assignment

(0) A. Quad_Beat_Tracks_Building_Base_Rhythm.mp3 (base)
(1) B. Quad_Beat_Tracks_Building_Building_Tracks-000.mp3 (inversion)
(2) C. Quad_Beat_Tracks_Building_Building_Tracks-001.mp3 (diag symmetry)
(3) D. Quad_Beat_Tracks_Building_Building_Tracks-002.mp3 (lat symmetry)
(4) E. Quad_Beat_Tracks_Building_Building_Tracks-003.mp3 (1 x parallel)
(5) F. Quad_Beat_Tracks_Building_Building_Tracks-004.mp3 (2 x parallel)
(6) G. Quad_Beat_Tracks_Building_Building_Tracks-005.mp3 (sides equal)
(7) H. Quad_Beat_Tracks_Building_Building_Tracks-006.mp3 (angles equal)

Each shape will have one or more tracks playing. Let's assign them like this for now (and adjust later for

  • Convex: A
  • Concave: B
  • Dart: B, C
  • Kite: A, C
  • Trapezoid: A, E
  • Isosceles Trapezoid: A, E, D
  • Parallelogram: A, F
  • Rhombus: A, C, F, G
  • Rectangle: A, D, F, H
  • Square: A, C, D, F, G, H

(@jessegreenberg, if while implementing you find swapping tracks above makes for better clarity/groovier beats, go for it! They are not mapped to particular elements at the moment. Just assigned based on increasing order).

2. Tracks - Volume Emphasis (Shape/State emphasis)

Here we'll try just giving each shape a part of the whole. Play all tracks all the time, but raise the volume of the assigned track (or lower the volume of all the rest, whichever works). I'm not sure how much to lower or raise, but probably something to quickly feel out during implementation.

We'll use the same letter assignments for the tracks as the previous set.

For the purposes of testing (without explicit mapping) - Let's order the shapes this way and just map on the above-ordered sound files. I realize there are actually 10 shapes and only 8 files for many of these. Let's skip Concave and Dart for now in this one.

  • Convex - A
  • Concave - (none)
  • Dart - (none)
  • Kite - B
  • Trapezoid - C
  • Isosceles Trapezoid - D
  • Parallelogram - E
  • Rhombus - F
  • Rectangle - G
  • Square - H

3. Melody (Shape/State Emphasis)

Similar to the last set, but instead of changing volume, we'll always play the base rhythm and swap out the tracks for each shape, so there is only ever two tracks playing (base + mapped track).

Again, there are not as many files as shapes here (6 of 10), so we'll have to cut a few for demo purposes.

ALWAYS: Quad_Melody_Tracks_Base_Music.mp3
Concave & Convex: Nothing additional
Kite & Dart: Quad_Melody_Tracks_000.mp3
Trapezoid & Isosceles: Quad_Melody_Tracks-005.mp3
Parallelogram: Quad_Melody_Tracks-004.mp3
Rectangle: Quad_Melody_Tracks-002.mp3
Rhombus: Quad_Melody_Tracks-003.mp3
Square: Quad_Melody_Tracks-001.mp3

4. Arpeggio (Hybrid Sequential/Track Shape Emphasis)

Here I believe the goal was to assign both an arpeggio that built up the higher farther into the shape hierarchy you go. I'll be honest that I have difficulty mapping this correctly since there isn't a linear mapping or "logical progression" from Convex to Square. We only have 8 files for 10 shapes that we can detect, so we'll skip Concave/Convex. Also, there are additional tracks to layer, but there are only 4 that I can see. Unless we have more, let's try just differentiating the parallelograms as a test?

Let's try building up from arpeggio files 000 - 007 in the following order:

  • Convex (none)
  • Concave (none)
  • Dart 000
  • Kite 000 to 001
  • Trapezoid 000 to 002
  • Isosceles Trapezoid 000 to 003
  • Parallelogram: 000 to 004, also play Quad_Building_Up_Arpeggio_Optional_Shape_Identifier_Tracks-000.mp3
  • Rhombus: 000 to 005, also play Quad_Building_Up_Arpeggio_Optional_Shape_Identifier_Tracks-001.mp3
  • Rectangle: 000 to 006, also play Quad_Building_Up_Arpeggio_Optional_Shape_Identifier_Tracks-002.mp3
  • Square: 000 to 007, also play Quad_Building_Up_Arpeggio_Optional_Shape_Identifier_Tracks-003.mp3

@jessegreenberg, over to you. Feel free to slack with questions/concerns!

@brettfiedler
Copy link
Member Author

@Ashton-Morris , just to note that there tend to be fewer tracks than shapes for the above sets. I have some alterations above to account for this, but we'll likely want to fill in the gaps before interviews.

@jessegreenberg
Copy link
Contributor

OK everything is implemented with the above commits with the exception of a few sound files in the "Arpeggio" design. I don't see any of the "Quad_Building_Up_Arpeggio_Optional_Shape_Identifier_Tracks" in the google doc file. Once those are in this will be ready for review.

jessegreenberg added a commit that referenced this issue Aug 2, 2022
@jessegreenberg
Copy link
Contributor

Since we don't have the "optional arpeggio" sounds we will try playing the melody sounds under the arpeggio, at a reduced volume with this design

Convex (none)
Concave (none)
Dart 000, also play Quad_Melody_Tracks_000.mp3
Kite 000 to 001, also play Quad_Melody_Tracks_000.mp3
Trapezoid 000 to 002, also play Quad_Melody_Tracks-005.mp3
Isosceles Trapezoid 000 to 003, also play Quad_Melody_Tracks-005.mp3
Parallelogram: 000 to 004, also play Quad_Melody_Tracks-004.mp3
Rhombus: 000 to 005, also play Quad_Melody_Tracks-003.mp3
Rectangle: 000 to 006, also play Quad_Melody_Tracks-002.mp3
Square: 000 to 007, also play Quad_Melody_Tracks-001.mp3

@jessegreenberg
Copy link
Contributor

Adjust track volumes according to graphic in previous post.

@BLFiedler are those the values I should use for both sound designs?

@brettfiedler
Copy link
Member Author

brettfiedler commented Dec 14, 2022

These apply only to Build up, though the overall output level should decrease for both sound designs.

A change to a couple of tracks are needed for Shape/Volume Emphasis. Keeping myself assigned for that task.

  • Assign new tracks for secondary sound design. @BLFiedler

@jessegreenberg
Copy link
Contributor

jessegreenberg commented Dec 14, 2022

#175 (comment) done, and set up so that the other sound design can have its own set of output level values when ready. Back to @BLFiedler for next steps.

@brettfiedler
Copy link
Member Author

brettfiedler commented Jan 23, 2023

For the secondary sound design, I swapped a few tracks for ones not being used that were a little less punchy, and differentiated Concave and Convex a bit more (moved concave to the simpler concave beat) in the above commit. I also restored the background volume to 0.15.

I wanted to make a few volume adjustment requests and then call this complete.

@jessegreenberg, I have a few mixing requests for when the sounds are foregrounded, but also one for the background.

  • 1.) Can you make individual adjustments to the following tracks when they are foregrounded:
    Related to this soundboard graphic:
    image

    • Index [0], convex, lower to 0.75 relative to maximum
    • Index [2], kite, lower to 0.80 '' ''
    • Index [6], rectangle, lower to 0.50 '' ''
  • 2.) The rectangle sound (track index 5), dominates a lot of the background tracks. I put the overall background volume back to "0.15", which I think works well for most of the tracks, but it'd be nice to cut track 5's by 50% in the background.

@jessegreenberg
Copy link
Contributor

Done. @BLFiedler can you please review?

@brettfiedler
Copy link
Member Author

Regardless of where it was, it sounds good to me now.

  • Can you make individual adjustments to the following tracks when they are foregrounded:

  • 2.) The rectangle sound (track index 5), dominates a lot of the background tracks.

Ack. I requested the wrong track. Meant 6 instead of 5. But, since you put in the structure @jessegreenberg, I could easily fix my mistake :) Done in above commit. I also raised the background volume of the trapezoid track.

I'm calling this design complete :dusts hands:

Offering to @Ashton-Morris to do a full review of the shape sounds in both designs (can be changed in the Preferences menu) before closing this issue.

@Ashton-Morris
Copy link

I just listened to them both: wipes tear from eye:

I think they sound beautiful. Feel free to close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants