Recording audio and aggregating via python vs bonsai #1212

matiasandina · 2022-12-30T16:44:24Z

matiasandina
Dec 30, 2022

I have a function that I use to record audio from a microphone and aggregate the amplitude on windows of 1 second.
I am thinking about integrating this function with my bonsai workflows. It might be that it's better to handle this from bonsai's audio capture and buffering and feed the array to python to perform the averaging (and likely possible to do everything in bonsai). Since I had the python code already working, I thought it might be possible to integrate it as a python node, and modify the function to be either triggered or to return to bonsai each time it ran so we get a proper timestamp. Any advice appreciated.

import pyaudio
import wave
import numpy as np
import keyboard
import sys
import time
from collections import deque

def read_microphone_amplitude(device_name, window_sec=1, save_interval_sec=60):
    p = pyaudio.PyAudio()
    # scan for device
    mic_index = []
    for index in range(p.get_device_count()):
        print(f"searching in device {index}")
        if p.get_device_info_by_index(index)['name'] == device_name:
            mic_index.append(index)
            print(f"Found {device_name} in device index {mic_index[0]}")
    
    if len(mic_index) > 1:
        print("More than one device found, likely error, using first one")
    # us the first one
    mic_index = mic_index[0]

    dev_rate = int(p.get_device_info_by_index(mic_index).get('defaultSampleRate'))
    print(f"Using default rate of {dev_rate}Hz")
    chunk_size = window_sec * dev_rate

    # Open a microphone stream
    channels = 1 if sys.platform == 'darwin' else 2
    stream = p.open(format=pyaudio.paInt16,
                    input_device_index = mic_index,
                    channels=channels,
                    rate=dev_rate,
                    input=True,
                    frames_per_buffer=chunk_size)

    # Create a deque to store the amplitudes
    amplitudes = deque(maxlen=save_interval_sec)
    start_time = time.time()
    # Read from the microphone in a continuous stream
    while True:
        data = stream.read(chunk_size)
        data = np.frombuffer(data, dtype=np.int16)
        amplitude = np.abs(data).mean()
        print(amplitude, end="\r")
        amplitudes.append(amplitude)

        # Save the amplitudes to a file every n iterations
        if len(amplitudes) == save_interval_sec:
            with open("amplitudes.txt", "a") as f:
                for amp in amplitudes:
                    f.write(str(amp) + "\n")
            amplitudes.clear()

        # For debug only
        if len(amplitudes)>10:
            end_time = time.time()
            print(end_time - start_time)
            break

        # Check for the "Esc" key
        if keyboard.is_pressed("esc"):
            break

    # Close the stream
    stream.stop_stream()
    stream.close()
    p.terminate()
    print(amplitudes)

    return amplitudes

def list_devices():
    p = pyaudio.PyAudio()
    # scan for device
    mic_index = []
    for index in range(p.get_device_count()):
        print(f"searching in device {index}")
        print(p.get_device_info_by_index(index)['name'])   


if __name__ == '__main__':
    list_devices()
    device_name = 'C922 Pro Stream Webcam: USB Audio (hw:1,0)'
    read_microphone_amplitude(device_name, window_sec=1)

While we are at it, I haven't fully tested having more than one microphone (e.g., 1 microphone per animal box and n>1 animals) with either the python script or bonsai (in fact, if multiple devices detected I'm using the first one only). It would be desired that the solution works for more than one microphone stream. One thing about using more than one mic, besides having to find the name-index match, is that I would need them to be in sync and here bonsai would be nice instead of doing it in different python threads or different pyaudio streams.

bruno-f-cruz · 2023-01-02T12:51:20Z

bruno-f-cruz
Jan 2, 2023
Collaborator

Hi!
The following workflow will get you the audio data from your audio card and average each incoming buffer:

Workflow (Copy-Paste)

<?xml version="1.0" encoding="utf-8"?>
<WorkflowBuilder Version="2.7.1"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns:al="clr-namespace:Bonsai.Audio;assembly=Bonsai.Audio"
               xmlns:dsp="clr-namespace:Bonsai.Dsp;assembly=Bonsai.Dsp"
               xmlns="https://bonsai-rx.org/2018/workflow">
<Workflow>
  <Nodes>
    <Expression xsi:type="Combinator">
      <Combinator xsi:type="al:AudioCapture">
        <al:DeviceName>Microphone (Realtek High Defini</al:DeviceName>
        <al:SampleRate>44100</al:SampleRate>
        <al:SampleFormat>Mono16</al:SampleFormat>
        <al:BufferLength>1000</al:BufferLength>
      </Combinator>
    </Expression>
    <Expression xsi:type="Combinator">
      <Combinator xsi:type="dsp:Average" />
    </Expression>
    <Expression xsi:type="MemberSelector">
      <Selector>Val0</Selector>
    </Expression>
  </Nodes>
  <Edges>
    <Edge From="0" To="1" Label="Source1" />
    <Edge From="1" To="2" Label="Source1" />
  </Edges>
</Workflow>
</WorkflowBuilder>

There are a few ways to control the binning size, but in this case, the simplest is to set the BufferLength property in the AudioCapture operator to the duration you want to average. It will use the SamplingRate property to calculate the size of the buffer, in samples, that corresponds to the time duration you specified.

For your second question, you can add as many of these branches as you want, as long as they target different devices. So that should not be a problem. Now, when it comes to synchronization I don't think Bonsai will solve your problem, unfortunately. The audio card/drivers are controlling the acquisition at the end of the day, and I don't believe multiple audio cards will provide any guarantee of synchrony across them. You could benchmark the jitter across mics. The most precise way would be to inject some sort of square wave signal and measure the onset times across streams. You could also go less fancy and just use a very loud clap.

Good luck,
Bruno

0 replies

matiasandina · 2023-01-05T15:25:36Z

matiasandina
Jan 5, 2023
Author

Hello, thank you for the reply.
I'm using this sketch, which is basically yours with my microphone (I tried using the same Logitech cam and a 3.5 mm mic), but I get no waveform displayed in the AudioCapture visualizer (no text either when visualizing as text), and obviously no signal at Val0.

<?xml version="1.0" encoding="utf-8"?>
<WorkflowBuilder Version="2.7.0"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xmlns:al="clr-namespace:Bonsai.Audio;assembly=Bonsai.Audio"
                 xmlns:dsp="clr-namespace:Bonsai.Dsp;assembly=Bonsai.Dsp"
                 xmlns="https://bonsai-rx.org/2018/workflow">
  <Workflow>
    <Nodes>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="al:AudioCapture">
          <al:DeviceName>Microphone (KT USB Audio)</al:DeviceName>
          <al:SampleRate>44100</al:SampleRate>
          <al:SampleFormat>Mono16</al:SampleFormat>
          <al:BufferLength>44100</al:BufferLength>
        </Combinator>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="dsp:Average" />
      </Expression>
      <Expression xsi:type="MemberSelector">
        <Selector>Val0</Selector>
      </Expression>
    </Nodes>
    <Edges>
      <Edge From="0" To="1" Label="Source1" />
      <Edge From="1" To="2" Label="Source1" />
    </Edges>
  </Workflow>
</WorkflowBuilder>

I sometimes got this error, which is not very clear to me but I think it might be due to choosing the wrong fields (like stereo instead of mono)

7 replies

bruno-f-cruz Jan 5, 2023
Collaborator

Can you try to install this:
Visual C++ Redistributable for Visual Studio 2012
It sounds like Bonsai cannot find the native dependencies it is looking for to run OpenCV.

If that does not work, reinstall bonsai and try again.

matiasandina Jan 5, 2023
Author

Ok, after re-install of the Visual C++ and full bonsai update to version 2.7.2, I have a signal displayed in the visualizer. I set the buffer to 1000 ms so I get updating every 1 second and one average value. Following on what you said about multiple mics, I get it that independent sound cards would not be perfectly in sync, but since this is a somewhat coarse measurement, I was hoping to do something like this:

This is not working super well, as shown in the pic

After a bit of testing, I found a few issues

Mics would drop to zero
Mics would not start for a long time
Mics would not update for several seconds at a time (see pic below)

Dropping the sampling rate on both mics to 16000 Hz partially helped with this particular issue but still dropping >50% of samples
Mics would update giving the exact same mean
When I stop the sketch, bonsai sometimes crashes and exists (something that it doesn't do when using only one mic).

Any input on this?
I could also not zip them and have each have their own timestamp, not sure what's the best.

Also, I was hoping to maybe move from the absolute of the mean to something like RMS, in python it would be:

rootMeanSquare = numpy.sqrt(numpy.mean(data**2))
decibel = 20 * math.log10(rootMeanSquare)
```

bruno-f-cruz Jan 5, 2023
Collaborator

I would probably just use CombineLatest to get the latest buffer from any of the mics when any of the mics emits a buffer. That might help. Regarding dropping samples, not sure what could be going on. Is it specific to one of the mics? Or is it related to load (i.e. when more than x mics are connected)

You need to timestamp prior to the zip to have independent timestamping.

You can use ExpressionTransforms after your Average->Val0. Something like: Math.Sqrt(Math.Pow(it,2)) for the first one. Math.Log10(Math.Sqrt(Math.Pow(it,2))) for the second one (?).

Check the Math class for some examples.

matiasandina Jan 5, 2023
Author

CombineLatest is better than zip and yes I could add independent timestamps.
Using ExpressionTransforms is a bit more challenging than expected. I have to do the math in different nodes (first square, then average, then sqrt, then log). I would also probably need a conditional to check for log(0). These chains are not working as I have them, not sure why but neither Math.Pow(it, 2) nor Math.Sqrt(it) change the value given to them. The crashing could be related to load, bonsai is crashing now after every stop.

<?xml version="1.0" encoding="utf-8"?>
<WorkflowBuilder Version="2.7.1"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xmlns:al="clr-namespace:Bonsai.Audio;assembly=Bonsai.Audio"
                 xmlns:dsp="clr-namespace:Bonsai.Dsp;assembly=Bonsai.Dsp"
                 xmlns:rx="clr-namespace:Bonsai.Reactive;assembly=Bonsai.Core"
                 xmlns:scr="clr-namespace:Bonsai.Scripting.Expressions;assembly=Bonsai.Scripting.Expressions"
                 xmlns="https://bonsai-rx.org/2018/workflow">
  <Workflow>
    <Nodes>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="al:AudioCapture">
          <al:DeviceName>Microphone (KT USB Audio)</al:DeviceName>
          <al:SampleRate>16000</al:SampleRate>
          <al:SampleFormat>Mono16</al:SampleFormat>
          <al:BufferLength>1000</al:BufferLength>
        </Combinator>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="dsp:Abs" />
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="dsp:Average" />
      </Expression>
      <Expression xsi:type="MemberSelector">
        <Selector>Val0</Selector>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="al:AudioCapture">
          <al:DeviceName>Microphone (2- C922 Pro Stream </al:DeviceName>
          <al:SampleRate>16000</al:SampleRate>
          <al:SampleFormat>Mono16</al:SampleFormat>
          <al:BufferLength>1000</al:BufferLength>
        </Combinator>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="dsp:Abs" />
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="dsp:Average" />
      </Expression>
      <Expression xsi:type="MemberSelector">
        <Selector>Val0</Selector>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="rx:CombineLatest" />
      </Expression>
      <Expression xsi:type="scr:ExpressionTransform">
        <scr:Name>square</scr:Name>
        <scr:Description>Math.Pow(it,2)</scr:Description>
        <scr:Expression>it</scr:Expression>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="dsp:Average" />
      </Expression>
      <Expression xsi:type="scr:ExpressionTransform">
        <scr:Name>sqrt</scr:Name>
        <scr:Description>Math.Sqrt(it)</scr:Description>
        <scr:Expression>it</scr:Expression>
      </Expression>
      <Expression xsi:type="scr:ExpressionTransform">
        <scr:Name>square</scr:Name>
        <scr:Description>Math.Pow(it,2)</scr:Description>
        <scr:Expression>it</scr:Expression>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="dsp:Average" />
      </Expression>
      <Expression xsi:type="scr:ExpressionTransform">
        <scr:Name>sqrt</scr:Name>
        <scr:Description>Math.Sqrt(it)</scr:Description>
        <scr:Expression>it</scr:Expression>
      </Expression>
    </Nodes>
    <Edges>
      <Edge From="0" To="1" Label="Source1" />
      <Edge From="0" To="9" Label="Source1" />
      <Edge From="1" To="2" Label="Source1" />
      <Edge From="2" To="3" Label="Source1" />
      <Edge From="3" To="8" Label="Source1" />
      <Edge From="4" To="5" Label="Source1" />
      <Edge From="4" To="12" Label="Source1" />
      <Edge From="5" To="6" Label="Source1" />
      <Edge From="6" To="7" Label="Source1" />
      <Edge From="7" To="8" Label="Source2" />
      <Edge From="9" To="10" Label="Source1" />
      <Edge From="10" To="11" Label="Source1" />
      <Edge From="12" To="13" Label="Source1" />
      <Edge From="13" To="14" Label="Source1" />
    </Edges>
  </Workflow>
</WorkflowBuilder>

bruno-f-cruz Jan 5, 2023
Collaborator

The expression should go in the Expression property of the ExpressionTransform node. You are typing it in the Description attribute.

matiasandina · 2023-01-06T15:05:29Z

matiasandina
Jan 6, 2023
Author

Oh that's lame from me, sorry about that. Getting this error though [image: image.png] @bruno-f-cruz Image was not rendering correctly here so I put it again ![image](https://user-images.githubusercontent.com/7494967/212757408-b3303ba5-0ea0-4796-b1e7-53a815827117.png) El jue, 5 ene 2023 a las 18:54, brunocruz ***@***.***>) escribió:

…

The expression should go in the Expression property of the ExpressionTransform node. You are typing it in the Description attribute. — Reply to this email directly, view it on GitHub <#1212 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZF2N67U3BXVUI5EODDF7DWQ5NLNANCNFSM6AAAAAATNAF5UI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bonsai

Recording audio and aggregating via python vs bonsai #1212

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Bonsai

Recording audio and aggregating via python vs bonsai #1212

matiasandina Dec 30, 2022

Replies: 3 comments · 7 replies

bruno-f-cruz Jan 2, 2023 Collaborator

matiasandina Jan 5, 2023 Author

bruno-f-cruz Jan 5, 2023 Collaborator

matiasandina Jan 5, 2023 Author

bruno-f-cruz Jan 5, 2023 Collaborator

matiasandina Jan 5, 2023 Author

bruno-f-cruz Jan 5, 2023 Collaborator

matiasandina Jan 6, 2023 Author

matiasandina
Dec 30, 2022

Replies: 3 comments 7 replies

bruno-f-cruz
Jan 2, 2023
Collaborator

matiasandina
Jan 5, 2023
Author

bruno-f-cruz Jan 5, 2023
Collaborator

matiasandina Jan 5, 2023
Author

bruno-f-cruz Jan 5, 2023
Collaborator

matiasandina Jan 5, 2023
Author

bruno-f-cruz Jan 5, 2023
Collaborator

matiasandina
Jan 6, 2023
Author