Membrane Audio Mix Plugin

Plugin providing elements for mixing and interleaving raw audio frames.

It is a part of Membrane Multimedia Framework.

Installation

First, you need to install MAD on your system.

macOS

brew install mad

Ubuntu

sudo apt-get install libmad0-dev

Then, add the following line to your deps in mix.exs:

	{:membrane_audio_mix_plugin, "~> 0.16.2"}

and run mix deps.get.

Description

Elements operate only on raw audio (PCM), so some parser may be needed to precede them in a pipeline.

The elements expect to receive the same audio stream format on each input pad.

Audio Mixers' input pads can have offset - in the case of a regular mixer it tells how much silence should be added before the first sample. The live mixer will rewrite timestamps based on the value. Offset has to be positive.

Mixing and interleaving are tested only for integer audio formats.

Mixer

The Mixer adds samples from all pads. It has two strategies to deal with the overflow: scaling down waves and clipping.

Stream format can be additionally enforced by setting an element option (:stream_format)

Interleaver

This element joins several mono audio streams (with one channel) into one stream with interleaved channels.

If audio streams have different durations, all shorter streams are appended with silence to match the longest stream.

Each channel must be named by providing an input pad name and the channel layout using those names must be provided (see usage example).

All inputs have to be added before starting the pipeline and should not be changed during interleaver's work.

Stream format can be additionally enforced by setting an element option (:stream_format)

Live Mixer

The Live Mixer adds samples from all pads. It has two strategies to deal with the overflow: scaling down waves and clipping. The Live Mixer will produce audio in real time meaning every interval there will be sent a buffer of fixed duration.

In situations where an audio packet is late or doesn't come, the mixer will fill the gaps with silence

Usage Example

AudioMixer

defmodule Mixing.Pipeline do
  use Membrane.Pipeline

  @impl true
  def handle_init(_ctx, _options) do
    structure = [
      child({:file_src, 1}, %Membrane.File.Source{location: "/tmp/input_1.raw"})
      |> get_child(:mixer),

      child({:file_src, 2}, %Membrane.File.Source{location: "/tmp/input_2.raw"})
      |> via_in(:input, options: [offset: Membrane.Time.milliseconds(5000)])
      |> get_child(:mixer),

      child(:mixer, %Membrane.AudioMixer{
        stream_format: %Membrane.RawAudio{
          channels: 1,
          sample_rate: 16_000,
          sample_format: :s16le
        }
      })
      |> child(:converter, %Membrane.FFmpeg.SWResample.Converter{
        input_stream_format: %Membrane.RawAudio{channels: 1, sample_rate: 16_000, sample_format: :s16le},
        output_stream_format: %Membrane.RawAudio{channels: 2, sample_rate: 48_000, sample_format: :s16le}
      })
      |> child(:player, Membrane.PortAudio.Sink)
    ]

    {[spec: structure], %{}}
  end
end

AudioInterleaver

defmodule Interleave.Pipeline do
  use Membrane.Pipeline

  alias Membrane.File.{Sink, Source}

  @impl true
  def handle_init(_ctx, {path_to_wav_1, path_to_wav_2}) do
    structure = [
      child({:file, 1}, %Source{location: path_to_wav_1})
      |> child({:parser, 1}, Membrane.WAV.Parser)
      |> get_child(:interleaver),

      child({:file, 2}, %Source{location: path_to_wav_2})
      |> child({:parser, 2}, Membrane.WAV.Parser)
      |> get_child(:interleaver),


      child(:interleaver, %Membrane.AudioInterleaver{
        input_stream_format: %Membrane.RawAudio{
          channels: 1,
          sample_rate: 16_000,
          sample_format: :s16le
        },
        order: [:left, :right]
      })
      |> child(:file_sink, %Sink{location: "output.raw"})
    ]

    {[spec: structure], %{}}
  end
end

Copyright and License

Copyright 2021, Software Mansion

Licensed under the Apache License, Version 2.0