Proposal: DecodeConcatVideoData (input multiple files or streams) => Output: single webm file

[guest271314]

Proposal: DecodeConcatVideoData (input multiple files or streams) => Output: single webm file

Web Audio API provides the ability to decodeAudioData where the result is a single AudioBuffer. AudioBuffers can be concatenated into a single AudioBuffer, see merging / layering multiple ArrayBuffers into one AudioBuffer using Web Audio API

I'm not sure I totally understand your scenario - don't you want these to be playing simultaneously? (i.e. bass gets layered on top of the drums).

Your current code is trying to concatenate an additional audio file whenever you hit the button for that file. You can't just concatenate audio files (in their ENCODED form) and then run it through decode - the decodeAudioData method is decoding the first complete sound in the arraybuffer, then stopping (because it's done decoding the sound).

What you should do is change the logic to concatenate the buffer data from the resulting AudioBuffers (see below). Even this logic isn't QUITE what you should do - this is still caching the encoded audio files, and decoding every time you hit the button. Instead, you should cache the decoded audio buffers, and just concatenate it.

function startStop(index, name, isPlaying) {

    // Note we're decoding just the new sound
    context.decodeAudioData( bufferList[index], function(buffer){
        // We have a decoded buffer - now we need to concatenate it
        audioBuffer = buffer;

        if(!audioBuffer) {
            audioBuffer = buffer;
        }else{
            audioBuffer = concatenateAudioBuffers(audioBuffer, buffer);
        }

        play();
    })
}

function concatenateAudioBuffers(buffer1, buffer2) {
    if (!buffer1 || !buffer2) {
        console.log("no buffers!");
        return null;
    }

    if (buffer1.numberOfChannels != buffer2.numberOfChannels) {
        console.log("number of channels is not the same!");
        return null;
    }

    if (buffer1.sampleRate != buffer2.sampleRate) {
        console.log("sample rates don't match!");
        return null;
    }

    var tmp = context.createBuffer(buffer1.numberOfChannels, buffer1.length + buffer2.length, buffer1.sampleRate);

    for (var i=0; i<tmp.numberOfChannels; i++) {
        var data = tmp.getChannelData(i);
        data.set(buffer1.getChannelData(i));
        data.set(buffer2.getChannelData(i),buffer1.length);
    }
    return tmp;
};

which can be played back.

Concatenating multiple MediaStreams into a single resulting webm file using MediaRecorder is not necessarily straightforward, though is possible using canvas.captureStream() and AudioContext.createMediaStreamDestination() and MediaRecorder(), see MediaStream Capture Canvas and Audio Simultaneously
; How to use Blob URL, MediaSource or other methods to play concatenated Blobs of media fragments?; https://github.com/guest271314/MediaFragmentRecorder/blob/canvas-webaudio/MediaFragmentRecorder.html;
and/or MediaSource() https://github.com/guest271314/MediaFragmentRecorder/tree/master (there is a Chromium bug using this approach, see w3c/media-source#190).

This proposal is for an API which accepts an Array of either multiple static files (potentially having different encodings/file extensions) and/or MediaStreams and outputs a single webm file (as a Blob) potentially "transcoded" (The process of converting a media asset from one codec to another.) in sequence to a "Mezzanine" file (4.1 Create a Mezzanine File; see also Scalable Video Coding (SVC) Extension for WebRTC; w3c/mediacapture-record#4) file (that is seekable, see https://bugs.chromium.org/p/chromium/issues/detail?id=642012).

For example

(async() => {
  try {
    const webmVideo = await DecodeConcatVideoData([
      "file.webm#5,10"
      , video.captureStream()
      , canvas.captureStream()
      , audioContextDestination.stream.getAudioTracks()[0]
    ]).catch(e => {throw e});
    // console.log(webmVideo); // `Blob`
  } catch(e) {

  }
})();

[alvestrand]

I think this belongs in MediaRecorder, if anywhere; this API does not deal with files.

[guest271314]

@alvestrand While you are around, what is the specification compliant procedure to get this code (close to what this issue proposes) https://github.com/guest271314/MediaFragmentRecorder/blob/webrtc-replacetrack/MediaFragmentRecorder.html working at Firefox (https://bugzilla.mozilla.org/show_bug.cgi?id=1542616#c8)?

[guest271314]

@alvestrand That is, where two RTCPeerConnection instances are created by the user within the same scope why is isolating or labeling the MediaStreamTrack "remote" mandated by the specification? Following your suggestion re MediaRecorder can that API be extended to include the functionality of RTCRtpSender.replaceTrack() (e.g., MediaRecorder.replaceTrack(withTrack)) without explicitly using RTCPeerConnection (which is only used for the functionality of replaceTrack()).

[guest271314]

@alvestrand Presumptively https://github.com/web-platform-tests/wpt/blob/master/webrtc-identity/RTCPeerConnection-peerIdentity.html setIdentityProvider() (defined at Firefox, not at Chromium/Chrome) would remove the "remote" label from the MediaStreamTrack? If that is correct can you include a complete example in the specification https://www.w3.org/TR/webrtc-identity/ itself of how to achieve that?

[aboba]

Closing since this issue does not relate to Media Capture and Streams.

[guest271314]

Closing since this issue does not relate to Media Capture and Streams.

@aboba This issue does directly relate to Media Capture and Streams, as evidenced by the title of the issue

Proposal: DecodeConcatVideoData (input multiple files or streams) => Output: single webm file

and the body of the issue

This proposal is for an API which accepts an Array of either multiple static files (potentially having different encodings/file extensions) and/or MediaStreams and outputs a single webm file (as a Blob) potentially "transcoded"

If, in your view, the issue does not relate to Media Capture and Streams, how do reconcile the fact that the title and content of the proposal specifically describe capturing media streams and outputting a single media file?